Building Durable Payment Workflows with Temporal in Go: My Lessons from Engineering a UK Fintech's

Building Durable Payment Workflows with Temporal in Go: My Lessons from Engineering a UK Fintech's

January 29, 2026

Originally published on Medium

Building Durable Payment Workflows with Temporal in Go: My Lessons from Engineering a UK Fintech’s Production System

How we built a production payment system that handles SEPA, SWIFT, and domestic transfers with 7-day workflow durability, legacy integration, and zero-downtime deployments.

The Project

I was consulting on a team building payment feature for UK fintech company.

Our mission: Build a modern, cloud-native payment system compatible with decades-old legacy payment infrastructure.

The catch? Payments are unforgiving. A transfer that fails halfway through is worse than one that never started. And our legacy banking system communicated through IBM MQ with messages encoded in formats older than most of our team members.

This is the story of how we built it.

The Problem We Faced

On day one, the existing payment flow looked something like this:

User clicks "Send £500"
    → API receives request
    → Validate account
    → Check fraud
    → Debit account
    → Send to core banking
    → Wait for confirmation
    → Credit recipient
    → Send notification

Simple enough on paper. But in production:

  • Step 4 fails: Did the debit happen? Do we retry? Do we reverse it?
  • Step 5 times out: The message went to IBM MQ but we never got a response. Is the payment processing? Did it fail silently?
  • Step 6 takes 3 days: International SWIFT payments can take days. How do we track state across service restarts and deployments?

Our first attempt used a state machine with PostgreSQL. It worked — until it didn’t. We had payments stuck in PROCESSING for weeks, no visibility into what went wrong, and engineers manually fixing database records at 2 AM.

We needed something better.

Why Temporal

After evaluating several options (Cadence, Step Functions, hand-rolled sagas), we chose Temporal. The selling points:

  • Durable execution: Workflows survive process crashes, deployments, even cluster failures
  • Native Go SDK: Our backend was Go, so this was a natural fit
  • Signal-based coordination: Perfect for waiting on async responses from legacy systems
  • Built-in visibility: The Temporal UI shows exactly where every workflow is stuck
  • Versioning: Deploy new workflow code without breaking in-flight payments

Here’s the architecture we ended up with:

The Payment Workflow

Let me walk you through how we implemented a SEPA payment. In the UK, SEPA (Single Euro Payments Area) handles Euro-denominated transfers to European banks — common for businesses with EU suppliers.

The workflow definition looked like:

package workflow

import (
	"fmt"
	"time"

	"go.temporal.io/sdk/temporal"
	"go.temporal.io/sdk/workflow"
)

// SEPAPaymentInput contains all data needed to process a SEPA payment
type SEPAPaymentInput struct {
	PaymentID       string
	CustomerID      string
	SourceAccountID string
	DestinationIBAN string
	DestinationBIC  string
	Amount          int64  // Amount in minor units (cents)
	Currency        string // EUR for SEPA
	Reference       string
	BeneficiaryName string
}

// SEPAPaymentOutput is the result of a completed payment
type SEPAPaymentOutput struct {
	PaymentID      string
	Status         string
	TransactionRef string
	CompletedAt    time.Time
}

// SEPAPayment orchestrates the complete payment lifecycle
func SEPAPayment(ctx workflow.Context, input SEPAPaymentInput) (*SEPAPaymentOutput, error) {
	logger := workflow.GetLogger(ctx)
	logger.Info("Starting SEPA payment workflow", "paymentID", input.PaymentID)

	// Configure activity options with sensible defaults
	activityOptions := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Minute,
		RetryPolicy: &temporal.RetryPolicy{
			InitialInterval:    time.Second,
			BackoffCoefficient: 2.0,
			MaximumInterval:    time.Minute,
			MaximumAttempts:    5,
		},
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)

	// Step 1: Validate the payment request
	var validationResult ValidationResult
	err := workflow.ExecuteActivity(ctx, ValidatePayment, input).Get(ctx, &validationResult)
	if err != nil {
		return nil, fmt.Errorf("validation failed: %w", err)
	}

	// Step 2: Check if this payment requires approval
	if err := handleApprovalIfRequired(ctx, input); err != nil {
		return nil, err
	}

	// Step 3: Security checks - detect suspicious patterns
	blocked, err := checkSecurityAndBlock(ctx, input)
	if err != nil {
		return nil, err
	}
	if blocked {
		// Wait for manual review - workflow pauses here
		if err := waitForUnblock(ctx); err != nil {
			return nil, err
		}
	}

	// Step 4: Fraud evaluation (single attempt)
	fraudCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 30 * time.Second,
		RetryPolicy: &temporal.RetryPolicy{
			MaximumAttempts: 1, // No retries for fraud checks
		},
	})

	var fraudResult FraudCheckResult
	err = workflow.ExecuteActivity(fraudCtx, CheckFraud, input).Get(ctx, &fraudResult)
	if err != nil {
		return nil, fmt.Errorf("fraud check failed: %w", err)
	}

	if fraudResult.Rejected {
		return nil, temporal.NewNonRetryableApplicationError(
			"Payment rejected by fraud system",
			"FRAUD_REJECTED",
			nil,
		)
	}

	// Step 5: Check if user cancelled while we were processing
	if cancelled, _ := isPaymentCancelled(ctx, input.PaymentID); cancelled {
		return nil, temporal.NewNonRetryableApplicationError(
			"Payment cancelled by user",
			"USER_CANCELLED",
			nil,
		)
	}

	// Step 6: Mark payee as trusted for future payments
	_ = workflow.ExecuteActivity(ctx, MarkPayeeAsTrusted, input).Get(ctx, nil)

	// Step 7: Publish to legacy banking system via IBM MQ
	var mqResult MQPublishResult
	err = workflow.ExecuteActivity(ctx, PublishToLegacyBanking, input).Get(ctx, &mqResult)
	if err != nil {
		return nil, fmt.Errorf("failed to publish to legacy system: %w", err)
	}

	// Step 8: Wait for confirmation from legacy system (this is the magic)
	paymentResult, err := waitForLegacyConfirmation(ctx, input.PaymentID)
	if err != nil {
		return nil, err
	}

	logger.Info("SEPA payment completed successfully",
		"paymentID", input.PaymentID,
		"transactionRef", paymentResult.TransactionRef)

	return paymentResult, nil
}

Waiting for Legacy Responses

The most interesting part of our implementation is Step 8. The legacy banking system processes payments asynchronously — it might take seconds, hours, or even days for international transfers. Traditional architectures would poll a database or use callbacks with complex state tracking.

With Temporal, we simply wait:

// waitForLegacyConfirmation blocks the workflow until we receive
// a signal from the legacy system (via mq-listener)
func waitForLegacyConfirmation(ctx workflow.Context, paymentID string) (*SEPAPaymentOutput, error) {
	logger := workflow.GetLogger(ctx)

	// Setup signal channels
	paymentSignalChan := workflow.GetSignalChannel(ctx, SignalPaymentResult)
	transactionSignalChan := workflow.GetSignalChannel(ctx, SignalTransactionResult)

	var paymentResult PaymentSignal
	var transactionResult TransactionSignal
	paymentReceived := false
	transactionReceived := false

	// We might receive payment confirmation and transaction confirmation separately
	for !paymentReceived || !transactionReceived {
		selector := workflow.NewSelector(ctx)

		// Handle payment status signal
		selector.AddReceive(paymentSignalChan, func(ch workflow.ReceiveChannel, more bool) {
			ch.Receive(ctx, &paymentResult)
			logger.Info("Received payment signal", "status", paymentResult.Status)
			paymentReceived = true
		})

		// Handle transaction confirmation signal
		selector.AddReceive(transactionSignalChan, func(ch workflow.ReceiveChannel, more bool) {
			ch.Receive(ctx, &transactionResult)
			logger.Info("Received transaction signal", "ref", transactionResult.Reference)
			transactionReceived = true
		})

		// Add timeout - don't wait forever
		timer := workflow.NewTimer(ctx, 7*24*time.Hour) // 7 days max for SWIFT
		selector.AddFuture(timer, func(f workflow.Future) {
			logger.Warn("Payment confirmation timeout", "paymentID", paymentID)
		})

		selector.Select(ctx)

		// Check for errors in received signals
		if paymentReceived && paymentResult.Status == "FAILED" {
			return nil, temporal.NewNonRetryableApplicationError(
				paymentResult.ErrorMessage,
				"LEGACY_REJECTED",
				nil,
			)
		}
	}

	return &SEPAPaymentOutput{
		PaymentID:      paymentID,
		Status:         "COMPLETED",
		TransactionRef: transactionResult.Reference,
		CompletedAt:    workflow.Now(ctx),
	}, nil
}

Below is a summary of some temporal patterns used:

  • We have used workflow.GetLogger(ctx), which retrieves a deterministic logger tied to the workflow context that is replay safe. This logger is used for recording events like signal receipts, timeouts, or other important workflow milestones. (link to workflow replays). Ideally, this should only be used in long-running workflow code for logging to maintain determinism and integrate with Temporal’s observability tools. It should not be used in non-workflow code, such as activities or plain Go functions, where standard loggers like log or zap suffice.
  • We also use workflow.GetSignalChannel(ctx, signalName), which creates a receive channel for external signals such as SignalPaymentResult or SignalTransactionResult. This allows workflows to listen for asynchronous updates from external systems without polling, with the channel state persisted across workflow suspensions (link). Signal channels are perfect for workflows that need to react to unpredictable events, like approvals or confirmations, ensuring durability even if the workflow pauses.
  • For handling multiple asynchronous events, we use workflow.NewSelector(ctx) along with AddReceive, AddFuture, and Select. This lets us multiplex events like signals and timers, executing callbacks when any event fires, and provides deterministic concurrency similar to Go’s native select but safe for workflows (link). Selectors are ideal for complex orchestration, such as waiting for multiple signals or activities concurrently without busy loops or non-deterministic behavior. They are unnecessary for simple sequential logic, activities, or when using Future.Get suffices. Word of caution: Overusing selectors can make workflows harder to debug.
  • To handle timeouts or bounded waits, we use workflow.NewTimer(ctx, duration), which schedules a durable timer that persists even through outages. For example, a 7-day timer prevents the workflow from waiting indefinitely for signals (link). Timers are essential for workflow reliability, particularly for retries, scheduling, or waiting on external dependencies. Use these only for long er delays
  • For controlled workflow failures, we use temporal.NewNonRetryableApplicationError(message, type, details). This creates a custom error for non-retryable failures, such as a legacy system rejection, allowing the workflow to fail gracefully without triggering automatic retries (link). This should only be used in workflows (not activities) for business logic errors that should not retry, like invalid data or rejections, and allows attaching debugging details. For transient issues like network failures, retryable errors should be used instead.
  • Finally, to handle consistent timestamps in workflows, we use workflow.Now(ctx), which returns the current time from Temporal’s perspective. This is deterministic and avoids inconsistencies caused by local clocks, making it ideal for recording events like CompletedAt or other workflow milestones (link).

The Legacy Bridge: Connecting Two Worlds

The legacy banking system spoke a different language — literally. Messages were encoded in a proprietary mainframe format and transmitted via IBM MQ. We built a bridge service to handle the translation.

Transforming Messages for the Legacy System

The legacy system expected messages in a specific tab-delimited format with fixed-width fields. Here’s how we transformed our domain objects:

package transform

import (
	"fmt"
	"strings"
	"time"
)

// LegacyPaymentMessage represents the format expected by the legacy system
type LegacyPaymentMessage struct {
	OLBReference     string // Unique reference we generate
	AccountBranch    string // 6 digits
	AccountBasic     string // 8 digits
	AccountSuffix    string // 3 digits
	PayeeAccountIBAN string
	PayeeBIC         string
	PayeeName        string // 35 chars max
	Amount           string // In minor units, no decimals
	Currency         string
	PaymentDate      string // YYYYMMDD
	Reference1       string // 35 chars max
	Reference2       string // 35 chars max
	Reference3       string // 35 chars max
	IsSEPA           string // Y/N
	ChargeType       string // SHA/OUR/BEN
}

// ToLegacyFormat converts a payment to the legacy tab-delimited format
func ToLegacyFormat(payment SEPAPaymentInput, olbRef string) []byte {
	msg := LegacyPaymentMessage{
		OLBReference:     olbRef,
		AccountBranch:    extractBranch(payment.SourceAccountID),
		AccountBasic:     extractBasic(payment.SourceAccountID),
		AccountSuffix:    extractSuffix(payment.SourceAccountID),
		PayeeAccountIBAN: payment.DestinationIBAN,
		PayeeBIC:         payment.DestinationBIC,
		PayeeName:        truncate(payment.BeneficiaryName, 35),
		Amount:           fmt.Sprintf("%d", payment.Amount),
		Currency:         payment.Currency,
		PaymentDate:      time.Now().Format("20060102"),
		Reference1:       splitReference(payment.Reference, 0),
		Reference2:       splitReference(payment.Reference, 1),
		Reference3:       splitReference(payment.Reference, 2),
		IsSEPA:           "Y",
		ChargeType:       "SHA", // Shared charges for SEPA
	}
	return msg.Encode()
}

// Encode converts the message to tab-delimited legacy format
func (m LegacyPaymentMessage) Encode() []byte {
	fields := []string{
		m.OLBReference,
		m.AccountBranch,
		m.AccountBasic,
		m.AccountSuffix,
		m.PayeeAccountIBAN,
		m.PayeeBIC,
		padRight(m.PayeeName, 35),
		m.Amount,
		m.Currency,
		m.PaymentDate,
		padRight(m.Reference1, 35),
		padRight(m.Reference2, 35),
		padRight(m.Reference3, 35),
		m.IsSEPA,
		m.ChargeType,
	}
	return []byte(strings.Join(fields, "\t"))
}

// splitReference breaks a long reference into 35-char chunks
func splitReference(ref string, index int) string {
	const chunkSize = 35
	start := index * chunkSize
	if start >= len(ref) {
		return ""
	}
	end := start + chunkSize
	if end > len(ref) {
		end = len(ref)
	}
	return ref[start:end]
}

The MQ Publisher Service

The publisher was called to forward messages to IBM MQ:

package publisher

import (
	"context"
	"fmt"
	"time"

	"github.com/ibm-messaging/mq-golang/v5/ibmmq"
)

type MQPublisher struct {
	queueManager *ibmmq.MQQueueManager
	queue        ibmmq.MQObject
	config       Config
}

type Config struct {
	Host         string
	Port         int
	QueueManager string
	Channel      string
	Queue        string
	User         string
	Password     string
}

func NewMQPublisher(cfg Config) (*MQPublisher, error) {
	cd := ibmmq.NewMQCD()
	cd.ChannelName = cfg.Channel
	cd.ConnectionName = fmt.Sprintf("%s(%d)", cfg.Host, cfg.Port)

	csp := ibmmq.NewMQCSP()
	csp.AuthenticationType = ibmmq.MQCSP_AUTH_USER_ID_AND_PWD
	csp.UserId = cfg.User
	csp.Password = cfg.Password

	cno := ibmmq.NewMQCNO()
	cno.ClientConn = cd
	cno.SecurityParms = csp
	cno.Options = ibmmq.MQCNO_CLIENT_BINDING

	qMgr, err := ibmmq.Connx(cfg.QueueManager, cno)
	if err != nil {
		return nil, fmt.Errorf("failed to connect to queue manager: %w", err)
	}

	od := ibmmq.NewMQOD()
	od.ObjectType = ibmmq.MQOT_Q
	od.ObjectName = cfg.Queue

	openOptions := ibmmq.MQOO_OUTPUT | ibmmq.MQOO_FAIL_IF_QUIESCING

	queue, err := qMgr.Open(od, openOptions)
	if err != nil {
		qMgr.Disc()
		return nil, fmt.Errorf("failed to open queue: %w", err)
	}

	return &MQPublisher{
		queueManager: &qMgr,
		queue:        queue,
		config:       cfg,
	}, nil
}

// Publish sends a message to the legacy system with retry logic
func (p *MQPublisher) Publish(ctx context.Context, message []byte) error {
	return p.publishWithRetry(ctx, message, 3)
}

func (p *MQPublisher) publishWithRetry(ctx context.Context, message []byte, maxRetries int) error {
	var lastErr error

	for attempt := 0; attempt < maxRetries; attempt++ {
		if attempt > 0 {
			select {
			case <-ctx.Done():
				return ctx.Err()
			case <-time.After(5 * time.Second):
			}
		}

		err := p.put(message)
		if err == nil {
			return nil
		}
		lastErr = err

		// Check if it's a connection error that requires reconnect
		if mqErr, ok := err.(*ibmmq.MQReturn); ok {
			if mqErr.MQRC == ibmmq.MQRC_CONNECTION_BROKEN {
				if reconnectErr := p.reconnect(); reconnectErr != nil {
					return fmt.Errorf("reconnect failed: %w", reconnectErr)
				}
				continue
			}
		}
		return err
	}

	return fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}

func (p *MQPublisher) put(message []byte) error {
	md := ibmmq.NewMQMD()
	md.Format = ibmmq.MQFMT_STRING

	pmo := ibmmq.NewMQPMO()
	pmo.Options = ibmmq.MQPMO_NO_SYNCPOINT | ibmmq.MQPMO_NEW_MSG_ID

	return p.queue.Put(md, pmo, message)
}

The MQ Listener: Signaling Workflows

The listener was the other half of the bridge. It consumed messages from IBM MQ and signaled the appropriate Temporal workflows:

package listener

import (
	"context"
	"fmt"
	"time"

	"github.com/ibm-messaging/mq-golang/v5/ibmmq"
	"go.temporal.io/sdk/client"
)

type MQListener struct {
	queueManager   *ibmmq.MQQueueManager
	queue          ibmmq.MQObject
	temporalClient client.Client
	parser         *MessageParser
}

// Start begins listening for messages from the legacy system
func (l *MQListener) Start(ctx context.Context) error {
	logger := getLogger(ctx)
	logger.Info("Starting MQ listener")

	for {
		select {
		case <-ctx.Done():
			logger.Info("Shutting down MQ listener")
			return nil
		default:
			if err := l.processNextMessage(ctx); err != nil {
				logger.Error("Error processing message", "error", err)
				// Continue listening - don't crash on single message failure
			}
		}
	}
}

func (l *MQListener) processNextMessage(ctx context.Context) error {
	md := ibmmq.NewMQMD()
	gmo := ibmmq.NewMQGMO()
	gmo.Options = ibmmq.MQGMO_WAIT | ibmmq.MQGMO_FAIL_IF_QUIESCING
	gmo.WaitInterval = 10 * 1000 // 10 seconds

	buffer := make([]byte, 32768)
	datalen, err := l.queue.Get(md, gmo, buffer)
	if err != nil {
		if mqErr, ok := err.(*ibmmq.MQReturn); ok {
			if mqErr.MQRC == ibmmq.MQRC_NO_MSG_AVAILABLE {
				return nil // No message, this is normal
			}
		}
		return fmt.Errorf("failed to get message: %w", err)
	}

	message := buffer[:datalen]
	parsed, err := l.parser.Parse(message)
	if err != nil {
		return fmt.Errorf("failed to parse message: %w", err)
	}

	return l.signalWorkflow(ctx, parsed)
}

func (l *MQListener) signalWorkflow(ctx context.Context, msg ParsedMessage) error {
	var signalName string
	var signalData interface{}

	switch msg.Type {
	case MessageTypePaymentResult:
		signalName = "SignalPaymentResult"
		signalData = PaymentSignal{
			Status:       msg.Status,
			ErrorCode:    msg.ErrorCode,
			ErrorMessage: msg.ErrorMessage,
		}
	case MessageTypeTransactionConfirmation:
		signalName = "SignalTransactionResult"
		signalData = TransactionSignal{
			Reference:   msg.TransactionRef,
			CompletedAt: msg.Timestamp,
		}
	default:
		return fmt.Errorf("unknown message type: %s", msg.Type)
	}

	return l.signalWithRetry(ctx, msg.WorkflowID, signalName, signalData)
}

func (l *MQListener) signalWithRetry(ctx context.Context, workflowID, signalName string, data interface{}) error {
	maxRetries := 3

	for attempt := 0; attempt < maxRetries; attempt++ {
		err := l.temporalClient.SignalWorkflow(ctx, workflowID, "", signalName, data)
		if err == nil {
			return nil
		}

		// Check if workflow is already completed - that's OK
		if isWorkflowNotFoundError(err) {
			getLogger(ctx).Warn("Workflow not found, may already be completed",
				"workflowID", workflowID)
			return nil
		}

		if attempt < maxRetries-1 {
			time.Sleep(time.Second * time.Duration(attempt+1))
		}
	}

	return fmt.Errorf("failed to signal workflow after %d attempts", maxRetries)
}

Handling Bulk Payments with the Saga Pattern

One of our more complex requirements was bulk payments — businesses uploading CSV files with hundreds of payments to process. We couldn’t let a single failed payment bring down the entire batch.

The Challenge

CSV with 500 payments uploaded
    → Payment 1: Success
    → Payment 2: Success
    → ...
    → Payment 247: FAILS (insufficient funds)
    → What happens to payments 1-246?
    → What about 248-500?

The Solution: Saga with Compensation

package workflow

// BulkPaymentInput represents a batch of payments
type BulkPaymentInput struct {
	BatchID    string
	Payments   []SinglePaymentLine
	UploadedBy string
	UploadedAt time.Time
}

type SinglePaymentLine struct {
	LineNumber  int
	Reference   string
	PaymentData SEPAPaymentInput
}

type BulkPaymentResult struct {
	BatchID     string
	TotalLines  int
	Successful  int
	Failed      int
	FailedLines []FailedLine
}

type FailedLine struct {
	LineNumber int
	Reference  string
	Error      string
}

// BulkPaymentWorkflow processes a batch of payments with compensation
func BulkPaymentWorkflow(ctx workflow.Context, input BulkPaymentInput) (*BulkPaymentResult, error) {
	logger := workflow.GetLogger(ctx)
	logger.Info("Starting bulk payment workflow",
		"batchID", input.BatchID,
		"totalPayments", len(input.Payments))

	activityOptions := workflow.ActivityOptions{
		StartToCloseTimeout: 5 * time.Minute,
		RetryPolicy: &temporal.RetryPolicy{
			InitialInterval:    time.Second,
			BackoffCoefficient: 2.0,
			MaximumInterval:    100 * time.Second,
			MaximumAttempts:    5,
		},
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)

	result := &BulkPaymentResult{
		BatchID:     input.BatchID,
		TotalLines:  len(input.Payments),
		FailedLines: []FailedLine{},
	}

	// Process each payment individually
	for _, line := range input.Payments {
		err := processPaymentLine(ctx, input.BatchID, line)
		if err != nil {
			logger.Warn("Payment line failed",
				"lineNumber", line.LineNumber,
				"error", err)

			// Compensate: Mark line as failed so it can be retried later
			compensateErr := compensateFailedLine(ctx, input.BatchID, line)
			if compensateErr != nil {
				logger.Error("Compensation failed",
					"lineNumber", line.LineNumber,
					"error", compensateErr)
			}

			result.Failed++
			result.FailedLines = append(result.FailedLines, FailedLine{
				LineNumber: line.LineNumber,
				Reference:  line.Reference,
				Error:      err.Error(),
			})

			// Continue processing remaining lines - don't fail the batch
			continue
		}

		result.Successful++
	}

	logger.Info("Bulk payment workflow completed",
		"batchID", input.BatchID,
		"successful", result.Successful,
		"failed", result.Failed)

	return result, nil
}

func processPaymentLine(ctx workflow.Context, batchID string, line SinglePaymentLine) error {
	// Update line status to PROCESSING
	err := workflow.ExecuteActivity(ctx, UpdateLineStatus, UpdateLineStatusInput{
		BatchID:    batchID,
		LineNumber: line.LineNumber,
		Status:     "PROCESSING",
	}).Get(ctx, nil)
	if err != nil {
		return fmt.Errorf("failed to update line status: %w", err)
	}

	// Execute the actual payment
	var paymentResult PaymentLineResult
	err = workflow.ExecuteActivity(ctx, ExecutePaymentLine, line.PaymentData).Get(ctx, &paymentResult)
	if err != nil {
		return err
	}

	// Update line status to COMPLETED
	return workflow.ExecuteActivity(ctx, UpdateLineStatus, UpdateLineStatusInput{
		BatchID:       batchID,
		LineNumber:    line.LineNumber,
		Status:        "COMPLETED",
		TransactionID: paymentResult.TransactionID,
	}).Get(ctx, nil)
}

// compensateFailedLine reverts the line status so it can be retried
func compensateFailedLine(ctx workflow.Context, batchID string, line SinglePaymentLine) error {
	return workflow.ExecuteActivity(ctx, RevertLineStatus, RevertLineStatusInput{
		BatchID:    batchID,
		LineNumber: line.LineNumber,
		Status:     "FAILED",
	}).Get(ctx, nil)
}

Compensation doesn’t mean rollback. For payments, we can’t magically un-send money. Instead, compensation means:

  • Recording that the line failed
  • Preserving enough state to retry or investigate later
  • Not letting one failure cascade to others

Workflow Versioning: Deploying Without Fear

One of the scariest parts of our initial design discussions was: “How do we deploy new code when we have payments running for days?”

Temporal’s versioning solved this elegantly:

func SEPAPayment(ctx workflow.Context, input SEPAPaymentInput) (*SEPAPaymentOutput, error) {
	// ... skipped steps for brevity ...

	// We added activity tracking in a later release
	// Use versioning to handle in-flight workflows
	activityTrackingVersion := workflow.GetVersion(ctx,
		"activity-tracking-v1",  // Unique identifier for this change
		workflow.DefaultVersion, // Version 0: old code path
		1,                       // Version 1: new code path
	)

	if activityTrackingVersion >= 1 {
		// New path: Send status updates for real-time UI
		_ = workflow.ExecuteActivity(ctx, PublishStatusUpdate, StatusUpdate{
			PaymentID: input.PaymentID,
			Status:    "PROCESSING",
			Step:      "FRAUD_CHECK",
		}).Get(ctx, nil)
	}

	// ... continue with fraud check ...

	// Another versioned change: we switched from signal-based approvals
	// to direct API calls
	approvalVersion := workflow.GetVersion(ctx,
		"approval-api-v2",
		workflow.DefaultVersion,
		2,
	)

	if approvalVersion >= 2 {
		// V2: Direct approval check
		err := workflow.ExecuteActivity(ctx, CheckApprovalDirect, input).Get(ctx, nil)
		if err != nil {
			return nil, err
		}
	} else if approvalVersion == 1 {
		// V1: Signal-based approval (deprecated)
		err := waitForApprovalSignal(ctx, input.PaymentID)
		if err != nil {
			return nil, err
		}
	}

	// ... rest of workflow ...
}

How it works

The version is persisted in the workflow history, so even after code changes, old workflows execute exactly as they did originally.

Error Handling: Knowing When NOT to Retry

Not all errors are created equal. Retrying “network timeout” makes sense. Retrying “insufficient funds” is pointless. We categorized errors carefully:

package activities

import (
	"context"
	"errors"
	"fmt"

	"go.temporal.io/sdk/temporal"
)

// Custom error types for business logic failures
var (
	ErrInsufficientFunds  = errors.New("insufficient funds")
	ErrAccountFrozen      = errors.New("account frozen")
	ErrInvalidBeneficiary = errors.New("invalid beneficiary")
	ErrDuplicatePayment   = errors.New("duplicate payment detected")
	ErrFraudRejected      = errors.New("rejected by fraud system")
	ErrUserCancelled      = errors.New("cancelled by user")
)

// ValidatePayment checks if a payment can proceed
func ValidatePayment(ctx context.Context, input SEPAPaymentInput) (ValidationResult, error) {
	// Check account balance
	balance, err := accountsClient.GetBalance(ctx, input.SourceAccountID)
	if err != nil {
		// Network error - Temporal should retry
		return ValidationResult{}, fmt.Errorf("failed to get balance: %w", err)
	}

	if balance < input.Amount {
		// Business error - don't retry, it won't help
		return ValidationResult{}, temporal.NewNonRetryableApplicationError(
			fmt.Sprintf("Insufficient funds: available %d, required %d", balance, input.Amount),
			"INSUFFICIENT_FUNDS",
			ErrInsufficientFunds,
		)
	}

	// Check account status
	account, err := accountsClient.GetAccount(ctx, input.SourceAccountID)
	if err != nil {
		return ValidationResult{}, fmt.Errorf("failed to get account: %w", err)
	}

	if account.Status == "FROZEN" {
		return ValidationResult{}, temporal.NewNonRetryableApplicationError(
			"Account is frozen",
			"ACCOUNT_FROZEN",
			ErrAccountFrozen,
		)
	}

	// Validate IBAN format
	if !isValidIBAN(input.DestinationIBAN) {
		return ValidationResult{}, temporal.NewNonRetryableApplicationError(
			fmt.Sprintf("Invalid IBAN format: %s", input.DestinationIBAN),
			"INVALID_IBAN",
			ErrInvalidBeneficiary,
		)
	}

	return ValidationResult{Valid: true}, nil
}

// CheckFraud evaluates the payment for fraud risk
func CheckFraud(ctx context.Context, input SEPAPaymentInput) (FraudCheckResult, error) {
	result, err := fraudService.Evaluate(ctx, FraudRequest{
		CustomerID: input.CustomerID,
		Amount:     input.Amount,
		Recipient:  input.DestinationIBAN,
	})
	if err != nil {
		// Fraud service down - this is configured with MaxAttempts: 1
		// so we'll fail rather than proceed without fraud check
		return FraudCheckResult{}, fmt.Errorf("fraud service unavailable: %w", err)
	}

	if result.Decision == "REJECT" {
		// Fraud rejection is a result, not an error
		return FraudCheckResult{
			Rejected: true,
			Reason:   result.Reason,
		}, nil
	}

	return FraudCheckResult{
		Rejected: false,
		Score:    result.RiskScore,
	}, nil
}

Here is how the prod setup looked like:

package config

import (
	"time"

	"go.temporal.io/sdk/temporal"
	"go.temporal.io/sdk/worker"
)

const (
	PaymentTaskQueue = "payments-task-queue"

	// SWIFT payments can take days to settle
	WorkflowRunTimeout  = 7 * 24 * time.Hour
	WorkflowTaskTimeout = 10 * time.Minute

	DefaultActivityTimeout = 10 * time.Minute
	FraudCheckTimeout      = 30 * time.Second
	LegacyPublishTimeout   = 5 * time.Minute
)

// WorkerOptions returns production worker configuration
func WorkerOptions() worker.Options {
	return worker.Options{
		MaxConcurrentWorkflowTaskExecutionSize: 50,
		MaxConcurrentActivityExecutionSize:     100,
		MaxConcurrentWorkflowTaskPollers:       4,
		MaxConcurrentActivityTaskPollers:       8,
		StickyScheduleToStartTimeout:           5 * time.Second,
	}
}

// DefaultRetryPolicy for most activities
func DefaultRetryPolicy() *temporal.RetryPolicy {
	return &temporal.RetryPolicy{
		InitialInterval:    time.Second,
		BackoffCoefficient: 2.0,
		MaximumInterval:    time.Minute,
		MaximumAttempts:    5,
	}
}

// NoRetryPolicy for activities that shouldn't retry
func NoRetryPolicy() *temporal.RetryPolicy {
	return &temporal.RetryPolicy{
		MaximumAttempts: 1,
	}
}

Worker Setup:

package main

import (
	"log"
	"os"
	"os/signal"
	"syscall"

	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{
		HostPort:  os.Getenv("TEMPORAL_HOST"),
		Namespace: os.Getenv("TEMPORAL_NAMESPACE"),
	})
	if err != nil {
		log.Fatalf("Failed to create Temporal client: %v", err)
	}
	defer c.Close()

	w := worker.New(c, config.PaymentTaskQueue, config.WorkerOptions())

	// Register workflows
	w.RegisterWorkflow(workflow.SEPAPayment)
	w.RegisterWorkflow(workflow.SWIFTPayment)
	w.RegisterWorkflow(workflow.DomesticPayment)
	w.RegisterWorkflow(workflow.BulkPaymentWorkflow)
	w.RegisterWorkflow(workflow.StandingOrderWorkflow)

	// Register activities
	registerActivities(w)

	go func() {
		if err := w.Run(worker.InterruptCh()); err != nil {
			log.Fatalf("Worker failed: %v", err)
		}
	}()

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
	<-quit
	log.Println("Shutting down worker...")
	w.Stop()
}

func registerActivities(w worker.Worker) {
	accountAct := activities.NewAccountActivities(accountsClient)
	w.RegisterActivity(accountAct.GetAccount)
	w.RegisterActivity(accountAct.GetBalance)
	w.RegisterActivity(accountAct.DebitAccount)

	fraudAct := activities.NewFraudActivities(fraudClient)
	w.RegisterActivity(fraudAct.CheckFraud)

	mqAct := activities.NewMQActivities(mqPublisher)
	w.RegisterActivity(mqAct.PublishToLegacyBanking)

	validationAct := activities.NewValidationActivities(validationService)
	w.RegisterActivity(validationAct.ValidatePayment)

	statusAct := activities.NewStatusActivities(statusService)
	w.RegisterActivity(statusAct.UpdatePaymentStatus)
	w.RegisterActivity(statusAct.PublishStatusUpdate)
}

Summary of Lessons Learned

After running this in production, here’s a summary of what we learned:

1. Design Workflows Around Business Time, Not Computer Time

Our first instinct was to make workflows as short as possible. But payments don’t work that way — a SWIFT transfer might take 3 business days. We learned to embrace long-running workflows:

// DON'T: Arbitrary timeout that doesn't match business reality
WorkflowRunTimeout: 1 * time.Hour

// DO: Match the business process
WorkflowRunTimeout: 7 * 24 * time.Hour // International payments can take days

2. Signals Are Not Guaranteed Delivery

We initially assumed signals always reached their workflows. They don’t — if the workflow completes or fails before the signal arrives, it’s lost. We added idempotency checks:

err := temporalClient.SignalWorkflow(ctx, workflowID, "", signalName, data)
if err != nil {
	if isWorkflowNotFoundError(err) {
		// Workflow already completed - log and move on
		logger.Info("Workflow already completed, signal dropped",
			"workflowID", workflowID)
		return nil
	}
	return err
}

3. Version Early, Version Often

We added our first version check only after we had a bug in production. Now we add version checks proactively, for prominent changes:

v := workflow.GetVersion(ctx, "change-identifier", workflow.DefaultVersion, 1)
if v >= 1 {
	// New behavior
} else {
	// Old behavior (for in-flight workflows)
}

4. Non-Retryable Errors Need Clear Communication

When we return a non-retryable error, the workflow fails permanently. Users need to understand why:

// DON'T: Vague error
return temporal.NewNonRetryableApplicationError("failed", "ERROR", nil)

// DO: Clear, actionable error
return temporal.NewNonRetryableApplicationError(
	fmt.Sprintf("Payment rejected: Account %s has insufficient funds. "+
		"Available: £%.2f, Required: £%.2f",
		accountID, float64(balance)/100, float64(amount)/100),
	"INSUFFICIENT_FUNDS",
	ErrInsufficientFunds,
)

Building durable payment workflows with Temporal transformed how we approached distributed systems. The patterns we developed — signal-based coordination, saga compensation, workflow versioning, and the legacy bridge — are now foundational to our architecture.

The key insight isn’t about Temporal specifically. It’s about matching your technical architecture to your business reality. Payments are inherently long-running, cross multiple systems, and need to survive failures. Traditional request-response architectures fight this reality. Workflow orchestration embraces it.

If you’re building payment systems — or any long-running business process — I’d encourage you to explore workflow orchestration. The investment in understanding these patterns pays dividends in reliability, visibility, and developer sanity.