Skip to content

Package kernel/txtutil

Summary

This is the design document of kernel/txtutil package.

kernel/txtutil package provides text processing utilities.

Motivation

Text processing is one of the fundamental procession in any application including API Gateways. It can be used for HTTP header and body processing, generating contents and so on. It will be useful to isolate and share the utility of functions.

Usecase:

  • HTTP header value replace
  • HTTP body replace
  • HTTP response body generation
  • Static content generation

Goals

  • Provides generalized text processing functions.
    • Text matching functions.
    • Text replacing functions.
    • Handing template functions

Non-Goals

  • Add functions that would be used for specific purposes.

Technical Design

Matchers

Matcher functions provides string or []byte matching functions.

As an generalized interface, the matching functions have the following signature.

// MatchFunc[string] for string matching.
// MatchFunc[[]byte] for []byte matching.
type MatchFunc[T any] func(T) bool

Matcher interface have a method that satisfy the MatchFunc[T any].

// Matcher[string] for string matching.
// Matcher[[]byte] for []byte matching.
type Matcher[T any] interface {
  Match(T) bool
}

Supported matching types are listed below. Note that the match types can be added or removed in future updates.

Match type Used method Try on Go Playground
Exact == https://go.dev/play/p/tzXNQYFeEbm
Prefix strings#HasPrefix https://go.dev/play/p/f_eU7-K49ZV
Suffix strings#HasSuffix https://go.dev/play/p/dOpLnzu74nv
Contains strings#Contains https://go.dev/play/p/tbFRqJTL7vt
Path path#Match https://go.dev/play/p/xY56ZBgCGrO
FilePath filepath#Match https://go.dev/play/p/dHl5dof11ZF
Regex regexp#Regexp.Match https://go.dev/play/p/AjWEC9C_YIh
RegexPOSIX regexp#Regexp.Match https://go.dev/play/p/yP1LIgg0PAe

Replacers

Replacer functions provides string or []byte replacing functions.

As an generalized interface, the replace functions have the following signature.

// ReplaceFunc[string] for replacing string values.
// ReplaceFunc[[]byte] for replacing []byte values.
type ReplaceFunc[T any] func(T) T

Replacer interface have a method that satisfy the ReplaceFunc[T any].

// Replacer[string] for replacing string values.
// Replacer[[]byte] for replacing []byte values.
type Replacer[T any] interface {
  Replace(T) T
}

Supported replacing types are listed below. Note that the replace types can be added or removed in future updates.

Replace type Used method Try on Go Playground
Noop - https://go.dev/play/p/pEaudzyUeOX
Fixed - https://go.dev/play/p/gIBOBblqe4w
Value strings#ReplaceAll https://go.dev/play/p/Qz5wmj2MKca
Left strings#Repeat https://go.dev/play/p/_XnzkUO5DNE
Right strings#Repeat https://go.dev/play/p/nDmuJ1F-TX2
Trim strings#Trim https://go.dev/play/p/p-O6aMuV_R0
TrimLeft strings#TrimLeft https://go.dev/play/p/8tSm5oex608
TrimRight strings#TrimRight https://go.dev/play/p/xuH7DdM0yN_y
TrimPrefix strings#TrimPrefix https://go.dev/play/p/jhIFvHB8FoH
TrimSuffix strings#TrimSuffix https://go.dev/play/p/uY2WNAPlZ9M
Encode Regexp + Encode https://go.dev/play/p/yy1QvqHOjue
Hash Regexp + Hash https://go.dev/play/p/Hjc8Z2rR1wz
Regexp regexp#Regexp.ReplaceAll https://go.dev/play/p/T6Mu9usjstw
Expand regexp#Regexp.Expand https://go.dev/play/p/Pw6tLUVZYcO
Encrypt Regexp + Encryption https://go.dev/play/p/mBvv3DGInhg

Fasttemplate

Fasttemplate is the simple and very fast but limited feature template. It primarily intended to be used for

  • Error message formatting.
  • Log message formatting.

This feature is the replacement of valyala/fasttemplate. valyala/fasttemplate is a nice package but raises many panics because of the limited type support.

This feature has no generalized signatures or interfaces. Use the template struct directly. This is an overview usage of the template.

input := map[string]any{
  "foo": "alice",
  "bar": "bob",
}

// func(tpl string, start string, end string) *txtutil.FastTemplate
tpl := txtutil.NewFastTemplate(`Hello {{foo}} and {{bar}}!`, "{{", "}}")

fmt.Println(string(tpl.Execute(input))) //  Hello alice and bob!

Template

Template functions provides document template features. It contains a function that generate a content from template using external information.

Template is defined a the common interface for template content. Its Content method returns the content generated from the template. It accepts external, or embed-able, information.

type Template interface {
  Content(map[string]any) []byte
}

Following types of templates are defined. Note that the template types can be added or removed in future development.

Template type Description Embed values Used Package
Text Plain text template Not supported -
GoText Text template using go standard package Supported text/template
GoHTML HTML template using go standard package Supported html/template

For example,

template (this format is just an example)

{
  "status": {{ status }},
  "code": "{{ code }}",
  "message": "{{ message }}" 
}

and external information

map[string]any{
  "status": 500,
  "code": "E1234",
  "message": "cannot connect to the session storage",
}

will produce the following content.

{
  "status": 500,
  "code": "E1234",
  "message": "cannot connect to the session storage" 
}

Test Plan

Unit Tests

Unit tests are implemented and passed.

  • All functions and methods are covered.
  • Coverage objective 98%.

Integration Tests

Not planned.

e2e Tests

Not planned.

Fuzz Tests

Fuzz tests are supposed to be implemented for

  • functions that accepts []byte data.
  • functions that accepts string data.

Benchmark Tests

Not planned.

Chaos Tests

Not planned.

Future works

Not planned.

References