Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

tokenizer

package

Go to main page

Versions in this module

v1

v1.2.0

Mar 15, 2026

v1.1.0

Mar 14, 2026

Changes in this version

+ type BPETokenizer struct

+ func LoadFromJSON(path string) (*BPETokenizer, error)

+ func NewBPETokenizer(vocab map[string]int, merges []MergePair, special SpecialTokens, ...) *BPETokenizer

+ func (t *BPETokenizer) Decode(ids []int) (string, error)

+ func (t *BPETokenizer) Encode(text string) ([]int, error)

+ func (t *BPETokenizer) EncodeWithSpecialTokens(text string, addBOS bool, addEOS bool) ([]int, error)

+ func (t *BPETokenizer) GetID(token string) (int, bool)

+ func (t *BPETokenizer) GetToken(id int) (string, bool)

+ func (t *BPETokenizer) SetSentencePiece(enabled bool)

+ func (t *BPETokenizer) SetSpecialTokenStrings(tokens map[string]int)

+ func (t *BPETokenizer) SpecialTokens() SpecialTokens

+ func (t *BPETokenizer) VocabSize() int

+ type MergePair struct

+ Left string

+ Right string

+ type NormalizerFunc func(string) string

+ type SpecialTokens struct

+ BOS int

+ EOS int

+ PAD int

+ UNK int

type Tokenizer

+ GetID func(token string) (int, bool)

+ SpecialTokens func() SpecialTokens

+ VocabSize func() int

+ type WhitespaceTokenizer struct

+ func NewWhitespaceTokenizer() *WhitespaceTokenizer

+ func (t *WhitespaceTokenizer) AddToken(token string) int

+ func (t *WhitespaceTokenizer) Decode(ids []int) (string, error)

+ func (t *WhitespaceTokenizer) Encode(text string) ([]int, error)

+ func (t *WhitespaceTokenizer) GetID(token string) (int, bool)

+ func (t *WhitespaceTokenizer) GetToken(id int) (string, bool)

+ func (t *WhitespaceTokenizer) SpecialTokens() SpecialTokens

+ func (t *WhitespaceTokenizer) VocabSize() int

v0

v0.3.0

Aug 25, 2025

Changes in this version

type Tokenizer

+ func (t *Tokenizer) GetToken(id int) string

v0.2.0

Aug 22, 2025

v0.1.0

Aug 4, 2025

Changes in this version

+ type Tokenizer struct

+ func NewTokenizer() *Tokenizer

+ func (t *Tokenizer) AddToken(token string) int

+ func (t *Tokenizer) Decode(tokenIDs []int) string

+ func (t *Tokenizer) Encode(text string) []int