Documentation
¶
Overview ¶
Package workqueue contains an interface for a simple key workqueue abstraction.
Metrics ¶
The GCS implementation exports the following Prometheus metrics:
- workqueue_in_progress_keys: The number of keys currently being processed
- workqueue_queued_keys: The number of keys currently in the backlog
- workqueue_notbefore_keys: The number of keys waiting on a 'not before' time
- workqueue_max_attempts: The maximum number of attempts for any queued or in-progress task
- workqueue_task_max_attempts: The maximum number of attempts for a given task above 20
- workqueue_process_latency_seconds: The duration taken to process a key
- workqueue_wait_latency_seconds: The duration the key waited to start
- workqueue_added_keys: The total number of queue requests
- workqueue_deduped_keys: The total number of keys that were deduped
- workqueue_attempts_at_completion: The number of attempts for successfully completed tasks
- workqueue_dead_lettered_keys: The number of keys currently in the dead letter queue
- workqueue_time_to_completion_seconds: The time from first queue to final outcome (success or dead-letter). The metric captures the full lifecycle duration including all retry attempts and backoff delays.
All metrics include service_name and revision_name labels. Additional labels vary by metric.
Index ¶
- Constants
- Variables
- func GetRequeueDelay(err error) (time.Duration, bool)
- func NonRetriableError(err error, reason string) error
- func RegisterWorkqueueServiceServer(s grpc.ServiceRegistrar, srv WorkqueueServiceServer)
- func RequeueAfter(delay time.Duration) error
- type Client
- type DeadLetteredKey
- type GetKeyStateRequest
- type InProgressKey
- type Interface
- type Key
- type KeyState
- func (*KeyState) Descriptor() ([]byte, []int)deprecated
- func (x *KeyState) GetAttempts() int32
- func (x *KeyState) GetKey() string
- func (x *KeyState) GetNotBeforeTime() int64
- func (x *KeyState) GetPriority() int64
- func (x *KeyState) GetQueuedTime() int64
- func (x *KeyState) GetStatus() KeyState_Status
- func (*KeyState) ProtoMessage()
- func (x *KeyState) ProtoReflect() protoreflect.Message
- func (x *KeyState) Reset()
- func (x *KeyState) String() string
- type KeyState_Status
- func (KeyState_Status) Descriptor() protoreflect.EnumDescriptor
- func (x KeyState_Status) Enum() *KeyState_Status
- func (KeyState_Status) EnumDescriptor() ([]byte, []int)deprecated
- func (x KeyState_Status) Number() protoreflect.EnumNumber
- func (x KeyState_Status) String() string
- func (KeyState_Status) Type() protoreflect.EnumType
- type NoRetryDetails
- type ObservedInProgressKey
- type Options
- type OwnedInProgressKey
- type ProcessRequest
- func (*ProcessRequest) Descriptor() ([]byte, []int)deprecated
- func (x *ProcessRequest) GetDelaySeconds() int64
- func (x *ProcessRequest) GetKey() string
- func (x *ProcessRequest) GetPriority() int64
- func (x *ProcessRequest) LogAttrs() []any
- func (*ProcessRequest) ProtoMessage()
- func (x *ProcessRequest) ProtoReflect() protoreflect.Message
- func (x *ProcessRequest) Reset()
- func (x *ProcessRequest) String() string
- type ProcessResponse
- type QueuedKey
- type UnimplementedWorkqueueServiceServer
- type UnsafeWorkqueueServiceServer
- type WorkqueueServiceClient
- type WorkqueueServiceServer
Examples ¶
Constants ¶
const ( WorkqueueService_Process_FullMethodName = "/chainguard.workqueue.WorkqueueService/Process" WorkqueueService_GetKeyState_FullMethodName = "/chainguard.workqueue.WorkqueueService/GetKeyState" )
Variables ¶
var ( // BackoffPeriod is the unit of backoff used when requeueing keys. // This unit is combined with the number of attempts to determine the // wait period before a key should be reprocessed. BackoffPeriod = 30 * time.Second // MaximumBackoffPeriod is a cap on the period a key must wait before // being retried. MaximumBackoffPeriod = 10 * time.Minute )
Note that these are variables, so that they can be modified by tests and made flags in binary entrypoints.
var ( KeyState_Status_name = map[int32]string{ 0: "UNKNOWN", 1: "QUEUED", 2: "IN_PROGRESS", 3: "DEAD_LETTER", } KeyState_Status_value = map[string]int32{ "UNKNOWN": 0, "QUEUED": 1, "IN_PROGRESS": 2, "DEAD_LETTER": 3, } )
Enum value maps for KeyState_Status.
var File_workqueue_proto protoreflect.FileDescriptor
var WorkqueueService_ServiceDesc = grpc.ServiceDesc{ ServiceName: "chainguard.workqueue.WorkqueueService", HandlerType: (*WorkqueueServiceServer)(nil), Methods: []grpc.MethodDesc{ { MethodName: "Process", Handler: _WorkqueueService_Process_Handler, }, { MethodName: "GetKeyState", Handler: _WorkqueueService_GetKeyState_Handler, }, }, Streams: []grpc.StreamDesc{}, Metadata: "workqueue.proto", }
WorkqueueService_ServiceDesc is the grpc.ServiceDesc for WorkqueueService service. It's only intended for direct use with grpc.RegisterService, and not to be introspected or modified (even as a copy)
Functions ¶
func GetRequeueDelay ¶ added in v0.6.161
GetRequeueDelay extracts the requeue delay from an error if it's a requeue error. Returns the delay and true if the error is a requeue error, or 0 and false otherwise.
func NonRetriableError ¶
NoRetryDetails marks the error as non-retriable with the given reason. If this error is returned to the dispatcher, it will not requeue the key.
func RegisterWorkqueueServiceServer ¶
func RegisterWorkqueueServiceServer(s grpc.ServiceRegistrar, srv WorkqueueServiceServer)
func RequeueAfter ¶ added in v0.6.161
RequeueAfter returns an error that indicates the work item should be requeued after the specified delay.
Example ¶
ExampleRequeueAfter demonstrates how to use RequeueAfter in a callback.
package main
import (
"context"
"fmt"
"time"
"github.com/chainguard-dev/terraform-infra-common/pkg/workqueue"
)
// ExampleWorker demonstrates how to implement a worker that can request
// custom requeue delays.
type ExampleWorker struct {
workqueue.UnimplementedWorkqueueServiceServer
}
func (w *ExampleWorker) Process(_ context.Context, req *workqueue.ProcessRequest) (*workqueue.ProcessResponse, error) {
// Example 1: Process successfully
if req.Key == "success" {
fmt.Println("Processing successful")
return &workqueue.ProcessResponse{}, nil
}
// Example 2: Requeue with a 5-minute delay
if req.Key == "rate-limited" {
fmt.Println("Rate limited, requeueing after 5 minutes")
return &workqueue.ProcessResponse{
RequeueAfterSeconds: 300, // 5 minutes
}, nil
}
// Example 3: Requeue with exponential backoff based on external state
if req.Key == "backoff" {
retryCount := getRetryCount(req.Key) // hypothetical function
delay := time.Duration(retryCount) * time.Minute
fmt.Printf("Requeueing with %v delay\n", delay)
return &workqueue.ProcessResponse{
RequeueAfterSeconds: int64(delay.Seconds()),
}, nil
}
// Example 4: Traditional error handling (uses default backoff)
if req.Key == "error" {
return nil, fmt.Errorf("processing failed")
}
// Example 5: Non-retriable error
if req.Key == "permanent-failure" {
return nil, workqueue.NonRetriableError(
fmt.Errorf("unrecoverable error"),
"Resource does not exist",
)
}
return &workqueue.ProcessResponse{}, nil
}
// ExampleRequeueAfter demonstrates how to use RequeueAfter in a callback.
func main() {
callback := func(_ context.Context, _ string, _ workqueue.Options) error {
// Do some work...
// Request requeue with a 30-second delay
return workqueue.RequeueAfter(30 * time.Second)
}
// This would be used in a dispatcher
err := callback(context.Background(), "example-key", workqueue.Options{})
delay, ok := workqueue.GetRequeueDelay(err)
fmt.Printf("Requeue requested: %v, delay: %v\n", ok, delay)
}
func getRetryCount(_ string) int {
// This is a placeholder for demonstration
return 1
}
Output: Requeue requested: true, delay: 30s
Types ¶
type Client ¶
type Client interface {
WorkqueueServiceClient
Close() error
}
func NewWorkqueueClient ¶
type DeadLetteredKey ¶ added in v0.10.0
type DeadLetteredKey interface {
Key
// GetFailedTime returns the time when the key was dead-lettered.
GetFailedTime() time.Time
// GetAttempts returns the number of attempts before the key was dead-lettered.
GetAttempts() int
}
DeadLetteredKey is a key that has been moved to the dead-letter queue after exceeding the maximum retry attempts.
type GetKeyStateRequest ¶ added in v0.6.190
type GetKeyStateRequest struct {
// The key to retrieve information for
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
// contains filtered or unexported fields
}
func (*GetKeyStateRequest) Descriptor
deprecated
added in
v0.6.190
func (*GetKeyStateRequest) Descriptor() ([]byte, []int)
Deprecated: Use GetKeyStateRequest.ProtoReflect.Descriptor instead.
func (*GetKeyStateRequest) GetKey ¶ added in v0.6.190
func (x *GetKeyStateRequest) GetKey() string
func (*GetKeyStateRequest) ProtoMessage ¶ added in v0.6.190
func (*GetKeyStateRequest) ProtoMessage()
func (*GetKeyStateRequest) ProtoReflect ¶ added in v0.6.190
func (x *GetKeyStateRequest) ProtoReflect() protoreflect.Message
func (*GetKeyStateRequest) Reset ¶ added in v0.6.190
func (x *GetKeyStateRequest) Reset()
func (*GetKeyStateRequest) String ¶ added in v0.6.190
func (x *GetKeyStateRequest) String() string
type InProgressKey ¶
type InProgressKey interface {
Key
// Requeue returns this key to the queue.
Requeue(context.Context) error
// RequeueWithOptions returns this key to the queue with custom options.
RequeueWithOptions(context.Context, Options) error
}
InProgressKey is a shared interface that all in-progress key types must implement.
type Interface ¶
type Interface interface {
// Queue adds an item to the workqueue.
Queue(ctx context.Context, key string, opts Options) error
// Enumerate returns:
// - a list of all of the in-progress keys,
// - a list of the next "N" keys in the queue (according to its configured ordering),
// - a list of all dead-lettered keys, or
// - an error if the workqueue is unable to enumerate the keys.
Enumerate(ctx context.Context) ([]ObservedInProgressKey, []QueuedKey, []DeadLetteredKey, error)
// Get retrieves the current state and metadata for a specific key.
Get(ctx context.Context, key string) (*KeyState, error)
}
Interface is the interface that workqueue implementations must implement.
type Key ¶
type Key interface {
// Name is the name of the key.
Name() string
// Priority is the priority of the key.
Priority() int64
}
Key is a shared interface that all key types must implement.
type KeyState ¶ added in v0.6.190
type KeyState struct {
// The key name
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
Status KeyState_Status `protobuf:"varint,2,opt,name=status,proto3,enum=chainguard.workqueue.KeyState_Status" json:"status,omitempty"`
// Priority of the key (if queued or in progress)
Priority int64 `protobuf:"varint,3,opt,name=priority,proto3" json:"priority,omitempty"`
// Number of attempts made (if in progress)
Attempts int32 `protobuf:"varint,4,opt,name=attempts,proto3" json:"attempts,omitempty"`
// Time when the key was first queued
QueuedTime int64 `protobuf:"varint,5,opt,name=queued_time,json=queuedTime,proto3" json:"queued_time,omitempty"`
// Time when the key should be processed (NotBefore)
NotBeforeTime int64 `protobuf:"varint,6,opt,name=not_before_time,json=notBeforeTime,proto3" json:"not_before_time,omitempty"`
// contains filtered or unexported fields
}
func (*KeyState) Descriptor
deprecated
added in
v0.6.190
func (*KeyState) GetAttempts ¶ added in v0.6.190
func (*KeyState) GetNotBeforeTime ¶ added in v0.6.190
func (*KeyState) GetPriority ¶ added in v0.6.190
func (*KeyState) GetQueuedTime ¶ added in v0.6.190
func (*KeyState) GetStatus ¶ added in v0.6.190
func (x *KeyState) GetStatus() KeyState_Status
func (*KeyState) ProtoMessage ¶ added in v0.6.190
func (*KeyState) ProtoMessage()
func (*KeyState) ProtoReflect ¶ added in v0.6.190
func (x *KeyState) ProtoReflect() protoreflect.Message
type KeyState_Status ¶ added in v0.6.190
type KeyState_Status int32
Current status of the key
const ( KeyState_UNKNOWN KeyState_Status = 0 KeyState_QUEUED KeyState_Status = 1 KeyState_IN_PROGRESS KeyState_Status = 2 KeyState_DEAD_LETTER KeyState_Status = 3 )
func (KeyState_Status) Descriptor ¶ added in v0.6.190
func (KeyState_Status) Descriptor() protoreflect.EnumDescriptor
func (KeyState_Status) Enum ¶ added in v0.6.190
func (x KeyState_Status) Enum() *KeyState_Status
func (KeyState_Status) EnumDescriptor
deprecated
added in
v0.6.190
func (KeyState_Status) EnumDescriptor() ([]byte, []int)
Deprecated: Use KeyState_Status.Descriptor instead.
func (KeyState_Status) Number ¶ added in v0.6.190
func (x KeyState_Status) Number() protoreflect.EnumNumber
func (KeyState_Status) String ¶ added in v0.6.190
func (x KeyState_Status) String() string
func (KeyState_Status) Type ¶ added in v0.6.190
func (KeyState_Status) Type() protoreflect.EnumType
type NoRetryDetails ¶
type NoRetryDetails struct {
Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"` // A message describing why the key should not be retried.
// contains filtered or unexported fields
}
NoRetryDetails is a marker message that indicates that the key should not be retried.
func GetNonRetriableDetails ¶
func GetNonRetriableDetails(err error) *NoRetryDetails
GetNonRetriableDetails extracts the NoRetryDetails from the error if it exists. If the error is nil or does not contain NoRetryDetails, it returns nil.
func (*NoRetryDetails) Descriptor
deprecated
func (*NoRetryDetails) Descriptor() ([]byte, []int)
Deprecated: Use NoRetryDetails.ProtoReflect.Descriptor instead.
func (*NoRetryDetails) GetMessage ¶
func (x *NoRetryDetails) GetMessage() string
func (*NoRetryDetails) ProtoMessage ¶
func (*NoRetryDetails) ProtoMessage()
func (*NoRetryDetails) ProtoReflect ¶
func (x *NoRetryDetails) ProtoReflect() protoreflect.Message
func (*NoRetryDetails) Reset ¶
func (x *NoRetryDetails) Reset()
func (*NoRetryDetails) String ¶
func (x *NoRetryDetails) String() string
type ObservedInProgressKey ¶
type ObservedInProgressKey interface {
InProgressKey
// IsOrphaned checks whether the key has been orphaned by it's owner.
IsOrphaned() bool
}
ObservedInProgressKey is a key that we have observed to be in progress, but that we are not the owner of.
type Options ¶
type Options struct {
// Priority is the priority of the key.
// Higher values are processed first.
Priority int64
// NotBefore is the earliest time that the key should be processed.
// When deduplicating, the oldest time is used.
NotBefore time.Time
// Delay is an optional duration to wait before processing the key.
// This is used when requeueing with a custom delay.
Delay time.Duration
}
Options is a set of options that can be passed when queuing a key.
type OwnedInProgressKey ¶
type OwnedInProgressKey interface {
InProgressKey
// Complete marks the key as successfully completed, and removes it from
// the in-progress key set.
Complete(context.Context) error
// Deadletter permanently removes this key from the queue, indicating it has
// failed after exceeding the maximum retry attempts.
Deadletter(context.Context) error
// GetAttempts returns the current attempt count for the key.
GetAttempts() int
// Context is the context of the process heartbeating the key.
Context() context.Context
}
OwnedInProgressKey is an in-progress key where we have initiated the work, and own until it completes either successfully (Complete), or unsuccessfully (Requeue or Fail).
type ProcessRequest ¶
type ProcessRequest struct {
// The key of the work item
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
// The (optional) priority of the work item, where higher numbers are processed first.
Priority int64 `protobuf:"varint,2,opt,name=priority,proto3" json:"priority,omitempty"`
// The (optional) delay in second to wait before processing the work item.
DelaySeconds int64 `protobuf:"varint,3,opt,name=delay_seconds,json=delaySeconds,proto3" json:"delay_seconds,omitempty"`
// contains filtered or unexported fields
}
func (*ProcessRequest) Descriptor
deprecated
func (*ProcessRequest) Descriptor() ([]byte, []int)
Deprecated: Use ProcessRequest.ProtoReflect.Descriptor instead.
func (*ProcessRequest) GetDelaySeconds ¶
func (x *ProcessRequest) GetDelaySeconds() int64
func (*ProcessRequest) GetKey ¶
func (x *ProcessRequest) GetKey() string
func (*ProcessRequest) GetPriority ¶
func (x *ProcessRequest) GetPriority() int64
func (*ProcessRequest) LogAttrs ¶ added in v0.6.167
func (x *ProcessRequest) LogAttrs() []any
LogAttrs returns a slice of attributes for logging purposes.
func (*ProcessRequest) ProtoMessage ¶
func (*ProcessRequest) ProtoMessage()
func (*ProcessRequest) ProtoReflect ¶
func (x *ProcessRequest) ProtoReflect() protoreflect.Message
func (*ProcessRequest) Reset ¶
func (x *ProcessRequest) Reset()
func (*ProcessRequest) String ¶
func (x *ProcessRequest) String() string
type ProcessResponse ¶
type ProcessResponse struct {
// Optional: If set, indicates the work item should be requeued after the specified delay.
// If not set or 0, the default behavior applies (success if no error, or exponential backoff on error).
// This field is only honored when the RPC returns successfully (no error).
RequeueAfterSeconds int64 `protobuf:"varint,1,opt,name=requeue_after_seconds,json=requeueAfterSeconds,proto3" json:"requeue_after_seconds,omitempty"`
// contains filtered or unexported fields
}
func (*ProcessResponse) Descriptor
deprecated
func (*ProcessResponse) Descriptor() ([]byte, []int)
Deprecated: Use ProcessResponse.ProtoReflect.Descriptor instead.
func (*ProcessResponse) GetRequeueAfterSeconds ¶ added in v0.6.161
func (x *ProcessResponse) GetRequeueAfterSeconds() int64
func (*ProcessResponse) ProtoMessage ¶
func (*ProcessResponse) ProtoMessage()
func (*ProcessResponse) ProtoReflect ¶
func (x *ProcessResponse) ProtoReflect() protoreflect.Message
func (*ProcessResponse) Reset ¶
func (x *ProcessResponse) Reset()
func (*ProcessResponse) String ¶
func (x *ProcessResponse) String() string
type QueuedKey ¶
type QueuedKey interface {
Key
// Start initiates processing of the key, returning an OwnedInProgressKey
// on success and an error on failure.
Start(context.Context) (OwnedInProgressKey, error)
}
QueuedKey is a key that is in the queue, waiting to be processed.
type UnimplementedWorkqueueServiceServer ¶
type UnimplementedWorkqueueServiceServer struct{}
UnimplementedWorkqueueServiceServer must be embedded to have forward compatible implementations.
NOTE: this should be embedded by value instead of pointer to avoid a nil pointer dereference when methods are called.
func (UnimplementedWorkqueueServiceServer) GetKeyState ¶ added in v0.6.190
func (UnimplementedWorkqueueServiceServer) GetKeyState(context.Context, *GetKeyStateRequest) (*KeyState, error)
func (UnimplementedWorkqueueServiceServer) Process ¶
func (UnimplementedWorkqueueServiceServer) Process(context.Context, *ProcessRequest) (*ProcessResponse, error)
type UnsafeWorkqueueServiceServer ¶
type UnsafeWorkqueueServiceServer interface {
// contains filtered or unexported methods
}
UnsafeWorkqueueServiceServer may be embedded to opt out of forward compatibility for this service. Use of this interface is not recommended, as added methods to WorkqueueServiceServer will result in compilation errors.
type WorkqueueServiceClient ¶
type WorkqueueServiceClient interface {
Process(ctx context.Context, in *ProcessRequest, opts ...grpc.CallOption) (*ProcessResponse, error)
GetKeyState(ctx context.Context, in *GetKeyStateRequest, opts ...grpc.CallOption) (*KeyState, error)
}
WorkqueueServiceClient is the client API for WorkqueueService service.
For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
func NewWorkqueueServiceClient ¶
func NewWorkqueueServiceClient(cc grpc.ClientConnInterface) WorkqueueServiceClient
type WorkqueueServiceServer ¶
type WorkqueueServiceServer interface {
Process(context.Context, *ProcessRequest) (*ProcessResponse, error)
GetKeyState(context.Context, *GetKeyStateRequest) (*KeyState, error)
// contains filtered or unexported methods
}
WorkqueueServiceServer is the server API for WorkqueueService service. All implementations must embed UnimplementedWorkqueueServiceServer for forward compatibility.