WP_Token_Map

Use this class in specific circumstances with a static set of lookup keys which map to a static set of transformed values. For example, this class is used to map HTML named character references to their equivalent UTF-8 values. This class works differently than code calling in_array() and other methods. It internalizes lookup logic and provides helper interfaces to optimize lookup and transformation. It provides a method for precomputing the lookup tables and storing them as PHP source code. All tokens and substitutions must be shorter than 256 bytes. Example: $smilies = WP_Token_Map::from_array( array( '8O' =&gt; '😯', ':(' =&gt; '🙁', ':)' =&gt; '🙂', ':?' =&gt; '😕', ) ); true === $smilies-&gt;contains( ':)' ); false === $smilies-&gt;contains( 'simile' ); '😕' === $smilies-&gt;read_token( 'Not sure :?.', 9, $length_of_smily_syntax ); 2 === $length_of_smily_syntax; <h2>Precomputing the Token Map.</h2> Creating the class involves some work sorting and organizing the tokens and their replacement values. In order to skip this, it's possible for the class to export its state and be used as actual PHP source code. Example: // Export with four spaces as the indent, only for the sake of this docblock. // The default indent is a tab character. $indent = ' '; echo $smilies-&gt;precomputed_php_source_table( $indent ); // Output, to be pasted into a PHP source file: WP_Token_Map::from_precomputed_table( array( "storage_version" =&gt; "6.6.0", "key_length" =&gt; 2, "groups" =&gt; "", "long_words" =&gt; array(), "small_words" =&gt; "8O\x00:)\x00:(\x00:?\x00", "small_mappings" =&gt; array( "😯", "🙂", "🙁", "😕" ) ) ); <h2>Large vs. small words.</h2> This class uses a short prefix called the "key" to optimize lookup of its tokens.This means that some tokens may be shorter than or equal in length to that key.Those words that are longer than the key are called "large" while those shorter than or equal to the key length are called "small." This separation of large and small words is incidental to the way this class optimizes lookup, and should be considered an internal implementation detail of the class. It may still be important to be aware of it, however. <h2>Determining Key Length.</h2> The choice of the size of the key length should be based on the data being stored in the token map. It should divide the data as evenly as possible, but should not create so many groups that a large fraction of the groups only contain a single token. For the HTML5 named character references, a key length of 2 was found to provide a sufficient spread and should be a good default for relatively large sets of tokens. However, for some data sets this might be too long. For example, a list of smilies may be too small for a key length of 2. Perhaps 1 would be more appropriate. It's best to experiment and determine empirically which values are appropriate. <h2>Generate Pre-Computed Source Code.</h2> Since the WP_Token_Map is designed for relatively static lookups, it can be advantageous to precompute the values and instantiate a table that has already sorted and grouped the tokens and built the lookup strings. This can be done with WP_Token_Map::precomputed_php_source_table(). Note that if there is a leading character that all tokens need, such as &amp; for HTML named character references, it can be beneficial to exclude this from the token map. Instead, find occurrences of the leading character and then use the token map to see if the following characters complete the token. Example: $map = WP_Token_Map::from_array( array( 'simple_smile:' =&gt; '🙂', 'sob:' =&gt; '😭', 'soba:' =&gt; '🍜' ) ); echo $map-&gt;precomputed_php_source_table(); // Output WP_Token_Map::from_precomputed_table( array( "storage_version" =&gt; "6.6.0", "key_length" =&gt; 2, "groups" =&gt; "si\x00so\x00", "long_words" =&gt; array( // simple_smile:[🙂]. "\x0bmple_smile:\x04🙂", // soba:[🍜] sob:[😭]. "\x03ba:\x04🍜\x02b:\x04😭", ), "short_words" =&gt; "", "short_mappings" =&gt; array() } ); This precomputed value can be stored directly in source code and will skip the startup cost of generating the lookup strings. See $html5_named_character_entities. Note that any updates to the precomputed format should update the storage version constant. It would also be best to provide an update function to take older known versions and upgrade them in place when loading into from_precomputed_table(). <h2>Future Direction.</h2> It may be viable to dynamically increase the length limits such that there's no need to impose them.The limit appears because of the packing structure, which indicates how many bytes each segment of text in the lookup tables spans. If, however, care were taken to track the longest word length, then the packing structure could change its representation to allow for that. Each additional byte storing length, however, increases the memory overhead and lookup runtime. An alternative approach could be to borrow the UTF-8 variable-length encoding and store lengths of less than 127 as a single byte with the high bit unset, storing longer lengths as the combination of continuation bytes. Since it has not been shown during the development of this class that longer strings are required, this update is deferred until such a need is clear.