text-to-topic-tokens

Documentation / tokenize/text-to-topic-tokens

Other

Token

Defined in: packages/ai-research-agent/src/tokenize/text-to-topic-tokens.js:4

Properties

Property	Type	Description	Defined in
termCategory	number	The category of the term	packages/ai-research-agent/src/tokenize/text-to-topic-tokens.js:5
uniqueness	number	The uniqueness score of the term	packages/ai-research-agent/src/tokenize/text-to-topic-tokens.js:6
term	string	The actual term or phrase	packages/ai-research-agent/src/tokenize/text-to-topic-tokens.js:7

Topics

convertTextToTokens()

function convertTextToTokens(phrase: string, options?: object): object[];

Defined in: packages/ai-research-agent/src/tokenize/text-to-topic-tokens.js:35

Convert Text Query to Topic Phrase Tokens

Returns a list of phrases that are found in Wiki Titles/ dictionary phrases World Model that match the input phrase, or just the single word if found. Search results will be more accurate if we infer likely phrases and search for those words occuring together and not just split into words and find frequency. Examples are "white house" or "state of the art" which should be searched as a phrase but would return different context if split into words. As Led Zeppelin famously put it: ♫ "'Cause you know sometimes words have two meanings."

Parameters

Parameter	Type	Description
phrase	string
options?	{ phrasesModel: any; typosModel: any; checkTypos: number; ignoreStopWords: number; checkRootWords: number; }
options.phrasesModel?	any	remote model
options.typosModel?	any	remote model
options.checkTypos?	number	check for typos
options.ignoreStopWords?	number	ignore 300+ overused words
options.checkRootWords?	number	check for word's root stem

Returns

object[]

Example

const result = convertTextToTokens("The president of the united states is in the white house", { phrasesModel, typosModel });
  console.log(result);

Author

ai-research-agent (2024)