Function tokenize

tokenize(
    src: string,
    lang: any[] | ShjLanguage,
    token: (arg0: string, arg1: ShjToken) => void,
): Promise<void>
Function
Find the tokens in the given code and call the given callback

tokenize
Parameters
- src: string
  The code
- lang: any[] | ShjLanguage
  The language of the code
- token: (arg0: string, arg1: ShjToken) => void
  The callback function this function will be given
  
  the text of the token
  
  the type of the token
Returns Promise<void>
- Defined in index.js:60