Quality Functions - 17.1

Transforms all half-width ASCII characters (single-byte) in an attribute to their full-width (double-byte) representation.

Syntax

ASCTOFULL (attribute)

where attribute is the attribute to be transformed.

Example

ASCTOFULL (Attribute1)

If Attribute1 contains 150, this returns １５０.

Transforms all full-width ASCII characters (double-byte) in an attribute to their half-width (single-byte) representation.

Syntax

ASCTOHALF (attribute)

where attribute is the attribute to be transformed.

Example

ASCTOHALF (Attribute1)

If Attribute1 contains １５０, this returns 150.

Transforms Chinese number symbols in an attribute to their Arabic decimal equivalents.

Syntax

CJKTOARABICNUM (attribute)

where attribute is the attribute to be transformed.

Example

CJKTOARABICNUM (Attribute1)

If Attribute1 contains 百五十, this returns 150.

Transforms half-width characters in an attribute to their full-width form. For Japan, this function automatically composes kana sound marks (dakuten and handakuten) appropriately. See Japanese full-width and half-width characters for details.

Syntax

CJKTOFULL (attribute)

where attribute is the attribute to be transformed.

Example

CJKTOFULL (Attribute1)

If Attribute1 contains Trillium, this returns Ｔｒｉｌｌｉｕｍ.

Transforms full-width characters in an attribute to their half-width form. For Japan, this function automatically decomposes kana sound marks (dakuten and handakuten) appropriately. See Japanese full-width and half-width characters for details.

Syntax

CJKTOHALF (attribute)

where attribute is the attribute to be transformed.

Example

CJKTOHALF (Attribute1)

If Attribute1 contains Ｔｒｉｌｌｉｕｍ, this returns Trillium.

Transforms Traditional Chinese characters in an attribute to their Simplified Chinese equivalent.

Syntax

CTOSIMPCHINESE (attribute)

where attribute is the attribute to be transformed.

Example

CTOSIMPCHINESE (Attribute1)

If Attribute1 contains 號, this returns 号.

Transforms Simplified Chinese characters in an attribute to their Traditional Chinese equivalent.

Syntax

CTOTRADCHINESE (attribute)

where attribute is the attribute to be transformed.

Example

CTOTRADCHINESE (Attribute1)

If Attribute1 contains 号, this returns 號.

Separates the value in an attribute into tokens and returns the deduped and delimited list of tokens. It performs the deduplication on the attribute value by searching the maximum number of tokens per phrase first, and repeats the search after decrementing the number of tokens per phrase by 1 each time. This process will continue until the number of tokens per phrase reaches the minimum specified.

Syntax

DEDUPE ("attribute", min, max, "separator")

where

• attribute is the attribute to be deduped.
• min is the minimum number of tokens that can comprise a phrase. Default is 1.
• max is the maximum number of tokens that can comprise a phrase. Default is 5.
• separator is token separator characters. Default is space (" ").

General Guidelines

• The search is case-sensitive. For example, "Car" and "car" are not duplicates. When the data is in mixed case, the DEDUPE function can be used with the UPPER or LOWER function.
• Duplicate phrases cannot extend across previously removed phrase(s) within a given record.
• Pay special attention to the logic for multi-token phrase processing outlined in Example 1. When a duplicate multi-token phrase is found, the original order of tokens in the attribute may not be maintained.

Guidelines for the Set Selection Utility

• The numerical values will be returned based on currently calculated precision and string formatting.
• Duplicate phrases cannot extend across boundaries of a given attribute for a given record within the set.
• The maximum length of the returned string will be the length of the input attribute times the number of records in the set. For example, if the input attribute has a length of 30 and there are 4 records in the set each with a full 30 character of data and no duplicates are found, the concatenation of the 4 records will be 120 characters. If this returned string is written back to a receiving attribute with a length less than 120 characters, the returned data will be truncated to the length of the receiving attribute.

Example 1 - multi-token phrases

DEDUPE(Attribute1,1, 3, " ")

Attribute1 contains: 'one way st two way st wrong way st one way st'

It searches for a duplicate from the beginning of the string for the longest phrase of tokens (3). Since there is a duplicate for "one way st" at the beginning, this phrase is added to the output, and the duplicate is removed from the search string.

Note: If a duplicate is not found, it moves over one token and looks for a duplicate from the second token.

Output: one way st

Remaining search string: two way st wrong way st

There are no more 3 token duplicates, so it searches for 2 token phrases next. It searches for a duplicate from the beginning of the remaining string, and if not found, it moves over one token and searches from the second token. Since there is a duplicate for "way st " the phrase is added to the output, and the duplicate is removed from the search string.

Output: one way st way st

Remaining search string: two wrong

Since there are no more duplicates, the remaining tokens are added to the final output.

Final output: one way st way st two wrong

Example 2 - single token phrases

DEDUPE(Attribute1,1,1, " ")

Attribute1 contains: 'one way st two way st wrong way st one way st'

Since the maximum number of token is set to 1, only single tokens are considered. It searches for duplicates from the beginning of the string. Since there is a duplicate for "one," this is added to the output, and the duplicate is removed from the search string.

Note: If a duplicate is not found, it moves over one token and looks for duplicates from the second token.

Output: one

Remaining search string: way st two way st wrong way st way st

Next, there are duplicates for "way " so the token is added to the output, and the duplicates are removed from the search string.

Output: one way

Remaining search string: st two st wrong st st

Next, there are duplicates for "st" so the phrase is added to the output, and the duplicates are removed from the search string.

Output: one way st

Remaining search string: two wrong

Non-duplicate tokens are added to the output as the search is moving through the string. Therefore "two" is added to the output.

Output: one way st two

Finally the remaining token is added to the output.

Final output: one way st two wrong

Transforms spacing form sound marks (dakuten and handakutens) in an attribute to combining form. Usually used before JCOMPOSE.

Syntax

JCOMBINE (attribute)

where attribute is the attribute to be transformed.

Example

JCOMBINE (Attribute1)

JCOMPOSE (Attribute1)

If Attribute1 contains シ゛ オコータ゛, this returns シ"オコータ".

If the sound marks cannot be merged with the preceding character (such as "ア"), they will be written out in hankaku in the output. If you need those sound marks to be in zenkaku in the output, use JSMARK after JCOMPOSE (JCOMBINE + JCOMPOSE + JSMARK). See Japanese Sound Marks for details.

Merges combining form sound marks (dakuten and handakutens) with the base characters to build dakuten characters. It is recommended to use JCOMBINE.

Syntax

JCOMPOSE (attribute)

where attribute is the attribute to be transformed.

Example

JCOMBINE (Attribute1)

JCOMPOSE (Attribute1)

If Attribute1 contains シ"オコータ", this returns ジオコーダ.

If the sound marks cannot be merged with the preceding character (such as "ア"), they will be written out in hankaku in the output. If you need those sound marks to be in zenkaku in the output, use JSMARK after JCOMPOSE (JCOMBINE + JCOMPOSE + JSMARK). See Japanese Sound Marks for details.

Separate combining form sound marks (dakuten and handakutens) from their base character. Usually used before JSMARK. See Japanese Sound Marks for details.

Syntax

JDECOMPOSE (attribute)

where attribute is the attribute to be transformed.

Example

JDECOMPOSE (Attribute1)

JSMARK (Attribute1)

If Attribute1 contains ジオコーダ, this returns シ" オコータ".

Transforms small size yo-on and soku-on in an attribute to its large equivalent.

Zenkaku

Large: あいうえおつやゆよわアイウエオツヤユヨワ

Small: ぁぃぅぇぉっゃゅょゎ ァィゥェォッャュョヮ

Hankaku

Large: ｱｲｳｴｵﾂﾔﾕﾖ

Small: ｧｨｩｪｫｯｬｭｮ

Syntax

JHIRAGANASTOL (attribute)

where attribute is the attribute to be transformed.

Example

JHIRAGANASTOL (Attribute1)

If Attribute1 contains マッチャー, this returns マツチヤー.

Transform hiragana and full-width katakana characters in an attribute to Hebon style romaji. See Romaji characters.

Syntax

JKANATOROMAN (attribute)

where attribute is the attribute to be transformed.

Example

JKANATOROMAN (Attribute1)

If Attribute1 contains じょうぞうしょ, this returns jouzousho.

Transforms romaji (Hebon) characters in an attribute to full-width katakana. See Romaji characters.

Syntax

JROMANTOKANA (attribute)

where attribute is the attribute to be transformed.

Example

JROMANTOKANA (Attribute1)

If Attribute1 contains toririamu, this returns トリリアム.

Transforms combining form sound marks (dakuten and handakutens) in an attribute to spacing mark form. Usually used after CJKTOFULL or JDECOMPOSE. See Japanese full-width and half-width characters for details.

Syntax

JSMARK (attribute)

where attribute is the attribute to be transformed.

Example

JDECOMPOSE (Attribute1)

JSMARK (Attribute1)

If Attribute1 contains シ" オコータ", this returns シ゛ オコータ゛.

Transforms full-width katakana characters in an attribute to hiragana. If you want to convert half-width katakana characters to hiragara, run CJKTOFULL first and run JTOHIRAGANA. See Japanese full-width and half-width characters for details.

Syntax

JTOHIRAGANA (attribute)

where attribute is the attribute to be transformed.

Example

JTOHIRAGANA (Attribute1)

If Attribute1 contains トリリアム, this returns とりりあむ.

Transforms hiragana characters in an attribute to full-width katakana. See Japanese full-width and half-width characters for details.

Syntax

JTOKATAKANA (attribute)

where attribute is the attribute to be transformed.

Example

JTOKATAKANA (Attribute1)

If Attribute1 contains とりりあむ, this returns トリリアム.

Transforms Korean Hangul characters in an attribute to their romanized forms.

Syntax

KTOROMAN (attribute)

where attribute is the attribute to be transformed.

Example

KTOROMAN (Attribute1)

If Attribute1 contains 대치동, this returns daech'idong.

Compares attributes and/or values and returns a match score based on the Relationship Linker Comparison routines and modifiers. All comparison routines are available for this usage.

Syntax

MATCH ("routine", attribute1 or "value1", attribute2 or "value2", "modifier")

where

• routine is the name of Relationship Linker Comparison routine.

Note: All routines are case-sensitive.

• attribute1 is the attribute whose value you want to compare.
• value1 is the value you want to compare.
• attribute2 is the attribute against which attribute1 is compared.
• value2 is the value against which value1 is compared.
• modifier is the name of the routine modifier.

The number of attributes, values, and modifiers used in the syntax varies depending on the comparison routine.

Examples

• MATCH("FRSTNAME", "Jo","Jo-Anne") returns a score of '90.'
• MATCH("DISTANCE", att1, att2, att3, att4, "MI", "10", "20", "30", "40", "50")
• MATCH("DIFFER", att1, att2, "[10][20][30][40][60]")

Returns a calculated distance between two latitude and longitude coordinates, based on the DISTANCE Relationship Linker routine. Distance is measured in kilometers (KM), miles (MI), or nautical miles (NM). Each coordinate is made up of two numbers, one for latitude and one for longitude.

This function is useful to create an expression in the Transformer to append the calculated distance in a new attribute or to use as part of a conditional statement.

Syntax

PROXIMITY(LAT1, LAT2, LON1, LON2, "measurement.00")

where

• LAT1 and LAT2 are latitude coordinates, either an attribute name or a string of eight digits preceded by a plus (+) or minus (-); for example, -15830940. Optionally you can enclose them in double quotes.
• LON1 and LON2 are longitude coordinates, either an attribute name or a string of nine digits preceded by a plus (+) or minus (-); for example, +043362490. Optionally you can enclose them in double quotes.
• measurement is the measurement type used, either MI (miles), NM (nautical miles), or KM (kilometers). Measurement must be in double quotes.
• (Optional) 00 is the number of digits following the decimal point. For example, "MI.00" returns miles to two decimal places and "KM.000" returns kilometers to three decimal places. Trailing zeros after the decimal point are ignored. For example, 1.50 becomes 1.5.

Examples

• PROXIMITY(+12345678, +52345778, +00100034, -00010035, "KM") returns 9170 KM.
• PROXIMITY("+18362871", "+18152254", "-066563498", "-067148548", "MI") returns 4227 MI.
• PROXIMITY(att1,att2,att3,att4,"NM") returns the distance in nautical miles between the coordinate values in the specified attributes.
• PROXIMITY( +40730357 , +40750546 , -073946604 , -073888527, "MI.00" ) returns 3.35 MI.