Skip to main content

text-to-topic-tokens

Documentation / tokenize/text-to-topic-tokens

Other​

Token​

Defined in: tokenize/text-to-topic-tokens.js:4

Properties​

PropertyTypeDescriptionDefined in

term

string

The actual term or phrase

tokenize/text-to-topic-tokens.js:7

termCategory

number

The category of the term

tokenize/text-to-topic-tokens.js:5

uniqueness

number

The uniqueness score of the term

tokenize/text-to-topic-tokens.js:6

Topics​

convertTextToTokens()​

function convertTextToTokens(phrase: string, options?: object): object[];

Defined in: tokenize/text-to-topic-tokens.js:35

Convert Text Query to Topic Phrase Tokens​

Returns a list of phrases that are found in Wiki Titles/ dictionary phrases World Model that match the input phrase, or just the single word if found. Search results will be more accurate if we infer likely phrases and search for those words occuring together and not just split into words and find frequency. Examples are "white house" or "state of the art" which should be searched as a phrase but would return different context if split into words. As Led Zeppelin famously put it: ♫ "'Cause you know sometimes words have two meanings."

Parameters​

ParameterTypeDescription

phrase

string

options?

{ checkRootWords: number; checkTypos: number; ignoreStopWords: number; phrasesModel: any; typosModel: any; }

options.checkRootWords?

number

check for word's root stem

options.checkTypos?

number

check for typos

options.ignoreStopWords?

number

ignore 300+ overused words

options.phrasesModel?

any

remote model

options.typosModel?

any

remote model

Returns​

object[]

Example​

const result = convertTextToTokens("The president of the united states is in the white house", { phrasesModel, typosModel });
console.log(result);

Author​

ai-research-agent (2024)