Function convertTextToTokens

  • convertTextToTokens(phrase, options?): {
        termCategory: number;
        uniqueness: number;
        term: string;
    }[]

  • Convert Text Query to Topic Phrase Tokens

    Returns a list of phrases that are found in Wiki Titles/ dictionary phrases World Model that match the input phrase, or just the single word if found. Search results will be more accurate if we infer likely phrases and search for those words occuring together and not just split into words and find frequency. Examples are "white house" or "state of the art" which should be searched as a phrase but would return different context if split into words. As Led Zeppelin famously put it: ♫ "'Cause you know sometimes words have two meanings."

    Parameters

    • phrase: string
    • Optionaloptions: {
          phrasesModel: any;
          typosModel: any;
          checkTypos: number;
          ignoreStopWords: number;
          checkRootWords: number;
      } = {}
      • phrasesModel: any

        remote model

      • typosModel: any

        remote model

      • checkTypos: number

        check for typos

      • ignoreStopWords: number

        ignore 300+ overused words

      • checkRootWords: number

        check for word's root stem

    Returns {
        termCategory: number;
        uniqueness: number;
        term: string;
    }[]

    Example

    const result = convertTextToTokens("The president of the united states is in the white house", { phrasesModel, typosModel });
      console.log(result);

    Author

    ai-research-agent (2024)

Settings

Member Visibility

On This Page

Convert Text Query to Topic Phrase Tokens