server

package
v0.0.0-...-7c0b2e9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 17, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package server implements the Registrar gRPC service, including endpoint snapshot management, change broadcasting, and external registry synchronization.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Broadcaster 

type Broadcaster struct {
	// contains filtered or unexported fields
}

Broadcaster fans out endpoint events to connected agent watch streams. Each watcher is identified by a string key (typically cluster/node) and receives events on a buffered channel. Watchers carrying a service filter are indexed by service, so an endpoint change fans out to that service's consumers only (demand-scoped distribution) instead of every node. A watcher too slow to keep up is forcibly resynced: its channel is closed, ending its WatchEndpoints stream, and the agent's reconnect receives a fresh (filtered) snapshot — it must never silently miss an event and serve stale endpoints until something else triggers a resync.

func NewBroadcaster 

func NewBroadcaster(log *slog.Logger, metrics *Metrics) *Broadcaster

NewBroadcaster creates a Broadcaster. metrics may be nil to disable instrumentation.

func (*Broadcaster) Broadcast 

func (b *Broadcaster) Broadcast(events []*registrarv1.WatchEndpointsResponse)

Broadcast sends each event to the watchers subscribed to its service (the service's consumers plus full watchers). Events are sent non-blocking; a watcher whose channel is full has already missed an event, so it is forced to resync: its channel is closed, which ends its WatchEndpoints stream, and the agent reconnects to receive a fresh filtered snapshot. The alternative — silently dropping the event — leaves the agent serving stale endpoints with nothing to correct it until its next reconnect for unrelated reasons.

Version stamping (#1203): the caller stamps the batch's version on its events, but each watcher receives it on the LAST event of the batch that watcher is sent, and on no other. The agent adopts every non-empty version as its resume token, and a stream cut between two events of a batch would otherwise leave it presenting the post-batch version while missing the rest of the batch -- which a reconnect then treats as current. Filtered watchers receive different subsets, so "last" is per watcher. The unversioned copies are built once per event, not per watcher.

func (*Broadcaster) MarkVersion 

func (b *Broadcaster) MarkVersion(current func() string) int

MarkVersion hands every watcher that is not already there the snapshot's current version (#1241), as a version-only SNAPSHOT_COMPLETE on its channel.

The version moves without a batch reaching a watcher in two ways: a store revision that changes no contents (a write-behind flush landing, a re-Put of an identical endpoint), and a change to a service outside a filtered watcher's filter. Its cache is current either way, but its resume token and its last_version gauge stay behind, so the store-to-agent lag reads as lag when nothing is stale.

Correctness is the rule for a batch's last event (#1203): a version is handed out only where the receiver holds everything it names. publishMu is taken for WRITING, so no publication is between its snapshot mutation and its broadcast: every batch the current version includes is already on every channel, ahead of this marker, and a watcher that overflowed on one of them has been closed. The version names the whole snapshot, and a filtered watcher holds it for every service of its filter, which is what every token means. A full channel is skipped, not force-resynced: a missing marker loses no event, and the next call retries.

The caller throttles: the syncer calls it once per sync cycle. It returns the number of markers sent.

func (*Broadcaster) Publish 

func (b *Broadcaster) Publish(mutate func() []*registrarv1.WatchEndpointsResponse)

Publish runs mutate -- which applies a batch to the snapshot and returns the batch's events, version-stamped -- and broadcasts the events, as one publication with respect to SubscribeWith (see publishMu). Every snapshot mutation whose events are broadcast must go through Publish, or a watch starting between the mutation and its broadcast receives the batch twice: once in its snapshot and once, carrying an older version, after it. Publications are serialized (see publishMu); mutate must not publish.

func (*Broadcaster) Subscribe 

func (b *Broadcaster) Subscribe(id string, services []string) <-chan *registrarv1.WatchEndpointsResponse

Subscribe registers a new watcher scoped to the given services (nil = full watch; empty = watch nothing) and returns a channel that will receive endpoint events. The caller must call Unsubscribe when done.

func (*Broadcaster) SubscribeWith 

func (b *Broadcaster) SubscribeWith(id string, services []string, start func() string) <-chan *registrarv1.WatchEndpointsResponse

SubscribeWith subscribes like Subscribe and then runs start (the watch's snapshot read), both with every publication excluded (see publishMu): the returned channel receives exactly the batches that start's snapshot does not contain. start must only read in-memory state -- the snapshot is sent after SubscribeWith returns, while new batches buffer on the channel -- and must not publish. It returns the version its initial exchange hands the client, which MarkVersion then treats as already sent. A buffer that overflows while the snapshot is still being sent closes the channel, which ends the stream with DataLoss as for any slow watcher.

func (*Broadcaster) Unsubscribe 

func (b *Broadcaster) Unsubscribe(id string, ch <-chan *registrarv1.WatchEndpointsResponse)

Unsubscribe removes a watcher and closes its channel. The channel returned by the matching Subscribe call must be passed so that a stale caller (whose subscription was already replaced by a reconnect with the same id) does not close or delete the newer subscription's channel.

func (*Broadcaster) WatcherCount 

func (b *Broadcaster) WatcherCount() int

WatcherCount returns the number of currently subscribed watchers.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds the registrar server's OTel instruments. All methods are nil-receiver-safe so the server runs unchanged when telemetry is disabled (pass a nil *Metrics).

These instruments exist to make missed updates and stale state observable: a nonzero dropped-events counter means at least one agent was force-resynced (or, before PR4, silently diverged), and the store/snapshot revision gauges (ObserveSnapshot) against the agents' aether.agent.registry.last_version are the stored -> in-place -> applied lag queries (docs/runbook.md, #1193).

func NewMetrics 

func NewMetrics(meter metric.Meter) (*Metrics, error)

NewMetrics registers the registrar server instruments on the given meter.

func (*Metrics) ObserveSnapshot 

func (m *Metrics) ObserveSnapshot(snap *Snapshot, store registry.RevisionedLister) error

ObserveSnapshot registers the asynchronous snapshot-identity instruments (#1193), read from snap (and store, when the backend has revisions) at each collection:

  • aether.registrar.store_revision: the newest store revision this replica has seen (its etcd watch or listing headers). etcd only.
  • aether.registrar.snapshot_revision: the store revision of the listing this replica's snapshot was last installed from. etcd only.
  • aether.registrar.snapshot.content_hash: the content hash as the gauge's VALUE (snapshotversion.ContentHashValue: the first 13 hex digits of the hash, an integer below 2^52, exact in a float64). One series per replica with no label that changes, so a content change moves the value of the same series and nothing lingers in Prometheus (#1329). Two replicas reporting the same snapshot_revision with a different value (and no write-behind intent pending) diverged: that is a bug. It is reported on every backend and in every version form (clean, dirty "<rev>+<hash>", "hash:<hash>"): it names the contents, whatever the revision says. On the kubernetes backend there is no snapshot_revision to compare at, and two replicas may legitimately differ for a moment (health derives from each replica's clock), so the value is for reading by hand there, not for alerting.

The hash is exported only as that value. Until chart 2.5.9 it was also the content_hash label of an info gauge (always 1), deprecated in chart 2.4.12: a label that changes with the contents starts a new series at every change, and Prometheus keeps returning the superseded one for its 5-minute lookback (OTLP has no staleness marker, #1322). Do not bring a label like it back.

All of them are observed in ONE callback from ONE Snapshot.State(), a single critical section: the revision and the hash of a collection always belong to the same contents. (The SDK stamps each instrument's data point separately, so their timestamps can differ by a few milliseconds within one export.)

A nil receiver or nil snap registers nothing; store may be nil (backend without revisions), in which case the revision gauges are not reported.

type Origin 

type Origin struct {
	// Revision is the store revision the listing was taken at (etcd header
	// revision); 0 when the backend has none, which makes the version
	// content-addressed.
	Revision int64
	// Overlaid reports that the listing was patched before installation (the
	// write-behind overlay shielded at least one pending intent), so the
	// contents are not exactly the store's at Revision.
	Overlaid bool
}

Origin describes where a replacement state came from.

type RegistrarServer 

type RegistrarServer struct {
	registrarv1.UnimplementedRegistrarServiceServer

	// Embed grpcserver.Server for gRPC lifecycle management.
	grpcserver.Server
	// contains filtered or unexported fields
}

RegistrarServer implements the RegistrarServiceServer gRPC interface. It delegates writes to an external registry and serves reads from a local snapshot. Change notifications are pushed to agents via the broadcaster.

func NewRegistrarServer 

func NewRegistrarServer(
	reg registry.Registry,
	snapshot *Snapshot,
	broadcaster *Broadcaster,
	address string,
	log *slog.Logger,
	metrics *Metrics,
	grpcOpts ...grpc.ServerOption,
) *RegistrarServer

NewRegistrarServer creates a RegistrarServer and registers it on the gRPC server. Additional gRPC server options (e.g., TLS credentials) can be passed via grpcOpts. metrics may be nil to disable instrumentation.

func (*RegistrarServer) GateOnSync 

func (s *RegistrarServer) GateOnSync(ch <-chan struct{})

GateOnSync makes snapshot-serving RPCs (WatchEndpoints, ListAllEndpoints) block until ch is closed (the syncer's first completed cycle). Nil leaves the RPCs ungated.

func (*RegistrarServer) ListAllConfig 

func (s *RegistrarServer) ListAllConfig(ctx context.Context, _ *registrarv1.ListAllConfigRequest) (*registrarv1.ListAllConfigResponse, error)

ListAllConfig returns the clusterset-wide config projections (proposal 026, multi-cluster config propagation). Agents pull this from the spoke registrar rather than reading the registry directly (the standing directive: agents don't talk to the store). When the backend has no cross-cluster config plane (kubernetes), the registry does not implement ConfigExporter and this returns an empty set — config stays cluster-local, which is the correct degenerate behaviour.

func (*RegistrarServer) ListAllEndpoints 

func (s *RegistrarServer) ListAllEndpoints(ctx context.Context, req *registrarv1.ListAllEndpointsRequest) (*registrarv1.ListAllEndpointsResponse, error)

ListAllEndpoints returns all endpoints from the local snapshot.

func (*RegistrarServer) RegisterEndpoint 

func (s *RegistrarServer) RegisterEndpoint(ctx context.Context, req *registrarv1.RegisterEndpointRequest) (*registrarv1.RegisterEndpointResponse, error)

RegisterEndpoint applies the change to the snapshot and broadcasts it immediately (discovery moves at watch latency), then writes the external registry — through the write-behind queue when enabled, so a failing external write can never make a serving pod invisible to the mesh (rev-68 roll regression). Without a queue (tests/legacy) the write is synchronous and failures fail the RPC.

func (*RegistrarServer) UnregisterEndpoint 

func (s *RegistrarServer) UnregisterEndpoint(ctx context.Context, req *registrarv1.UnregisterEndpointRequest) (*registrarv1.UnregisterEndpointResponse, error)

UnregisterEndpoint applies the removal to the snapshot and broadcasts it immediately, then writes the external registry (write-behind when enabled; see RegisterEndpoint).

func (*RegistrarServer) UseWriteBehind 

func (s *RegistrarServer) UseWriteBehind(q *WriteBehindQueue)

UseWriteBehind enables snapshot-first registry mutations through q.

func (*RegistrarServer) WatchEndpoints 

func (s *RegistrarServer) WatchEndpoints(req *registrarv1.WatchEndpointsRequest, stream grpc.ServerStreamingServer[registrarv1.WatchEndpointsResponse]) error

WatchEndpoints streams endpoint events to the agent. It subscribes to the broadcaster, then sends a full snapshot (unless the client's last_version names the current contents -- see Snapshot.WatchStart), then forwards the incremental events buffered since the subscription and every later one. A request filter scopes both the snapshot and the incremental events to the named services (demand-scoped distribution): the agent re-asserts its filter on every reconnect, and an unset filter preserves the full watch. A client whose filter grew sends partial_resume instead of last_version and, when its token names the current contents, receives only the added services' endpoints (#1239; see Snapshot.WatchStart).

type Resume 

type Resume int

Resume is the outcome of a watch start (Snapshot.WatchStart). 

const (
	// ResumeResend: the client's token does not name the current contents; it
	// gets the full snapshot and the catalog.
	ResumeResend Resume = iota
	// ResumeCurrent: the token is the current version; the marker alone.
	ResumeCurrent
	// ResumeRenamed: the token names the current contents under an older name
	// (same content hash, another revision or another form). No endpoint
	// events; the catalog and a marker carrying the CURRENT version, so the
	// client swaps in an identical catalog and adopts the fresh token.
	ResumeRenamed
)

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is a thread-safe, versioned in-memory store of all service endpoints. It supports computing diffs between states and applying incremental changes.

func NewSnapshot 

func NewSnapshot() *Snapshot

NewSnapshot creates an empty Snapshot at generation 0.

func (*Snapshot) Apply 

func (s *Snapshot) Apply(events []*registrarv1.WatchEndpointsResponse) (string, []*registrarv1.WatchEndpointsResponse)

Apply applies a set of events to the snapshot, updating it in place. The version moves only when the contents do; on a revisioned snapshot it becomes "<rev>+<hash>" until the next sync re-derives the contents from the store. It returns the new version string plus the service-catalog transitions the events caused (a service's endpoint count crossing 0<->1 emits SERVICE_ADDED/SERVICE_REMOVED): deriving transitions inside Apply makes the catalog impossible to desync from the endpoint data it summarizes. Transitions are unversioned; the caller stamps and broadcasts them with the batch.

func (*Snapshot) Diff 

func (s *Snapshot) Diff(newEndpoints map[string]map[registryv1.Service_Protocol][]*registryv1.ServiceEndpoint) []*registrarv1.WatchEndpointsResponse

Diff compares a new set of endpoints against the current snapshot and returns the events needed to transition from the current state to the new state. It does not modify the snapshot.

A caller that goes on to Replace with the same state must use DiffAndReplace instead — see the TOCTOU note there.

func (*Snapshot) DiffAndReplace 

func (s *Snapshot) DiffAndReplace(endpoints map[string]map[registryv1.Service_Protocol][]*registryv1.ServiceEndpoint) ([]*registrarv1.WatchEndpointsResponse, string, []*registrarv1.WatchEndpointsResponse)

DiffAndReplace computes the diff against the current contents and installs the new contents in ONE critical section, returning the diff events, the new version, and the catalog transitions.

The two halves must not be separate lock acquisitions (#772, S13). The sync loop is not the only writer: an agent's RegisterEndpoint lands via Apply on a gRPC handler goroutine. One arriving between a separate Diff and Replace is broadcast as ENDPOINT_ADDED by Apply and then silently erased by Replace, which installs a state computed before that endpoint existed — with no compensating REMOVED event, so every watcher keeps an endpoint the snapshot no longer has until a later sync cycle happens to re-derive it.

Today that hole is masked by a different component: Syncer.writeBehind overlays the pending intents onto the new state before this call. But then the invariant is held by the write-behind queue rather than by the snapshot, and the legacy constructor path leaves writeBehind nil (NewRegistrarServer without a queue), where nothing masks it. Holding the lock across both halves makes the snapshot self-consistent on its own.

func (*Snapshot) DiffAndReplaceAt 

func (s *Snapshot) DiffAndReplaceAt(endpoints map[string]map[registryv1.Service_Protocol][]*registryv1.ServiceEndpoint, origin Origin) ([]*registrarv1.WatchEndpointsResponse, string, []*registrarv1.WatchEndpointsResponse)

DiffAndReplaceAt is DiffAndReplace for a listing with a known origin: the store revision it was taken at and whether it was overlaid. The resulting version is "<rev>.<hash>" for a clean revisioned listing, "<rev>+<hash>" for an overlaid one, and "hash:<hash>" without a revision.

func (*Snapshot) FullSnapshotEvents 

func (s *Snapshot) FullSnapshotEvents(filter map[string]struct{}) ([]*registrarv1.WatchEndpointsResponse, string)

FullSnapshotEvents returns the current contents of the snapshot as a slice of FULL_SNAPSHOT events, plus the version they amount to. This is used to send the initial state to a new watcher.

The events carry NO version (#1203): the agent adopts every non-empty version it receives as its resume token, and a stream cut after the first of them would leave it presenting the full snapshot's version while holding a fraction of it. The version rides only on SNAPSHOT_COMPLETE, the one point at which the receiver holds everything it names.

A non-nil filter scopes the result to those service names (a nil filter is the unfiltered, cluster-wide snapshot): a demand-scoped watcher discards everything outside its filter anyway, so the out-of-scope events are skipped before their protos are ever built.

func (*Snapshot) GetAll 

func (s *Snapshot) GetAll(protocol registryv1.Service_Protocol) map[string][]*registryv1.ServiceEndpoint

GetAll returns all endpoints organized by service name. The caller receives a copy that is safe to mutate.

func (*Snapshot) GetAllWithVersion 

func (s *Snapshot) GetAllWithVersion(protocol registryv1.Service_Protocol) (map[string][]*registryv1.ServiceEndpoint, string)

GetAllWithVersion returns all endpoints and the current version.

func (*Snapshot) RemoveIPs 

func (s *Snapshot) RemoveIPs(serviceName string, ips []string) ([]*registrarv1.WatchEndpointsResponse, string, []*registrarv1.WatchEndpointsResponse)

RemoveIPs removes the given IPs of a service under every protocol the snapshot holds them, returning one ENDPOINT_REMOVED event per removed entry (carrying its protocol and stored endpoint), the new version, and the catalog transitions. An IP the snapshot does not hold yields no event (#1206).

UnregisterEndpointRequest names no protocol, but the snapshot and every agent cache key an endpoint by (service, protocol, ip): a REMOVED event without a protocol matches nothing on either side, so the removal must be resolved against what is stored, under the same lock that removes it.

func (*Snapshot) Replace 

func (s *Snapshot) Replace(endpoints map[string]map[registryv1.Service_Protocol][]*registryv1.ServiceEndpoint) (string, []*registrarv1.WatchEndpointsResponse)

Replace atomically replaces the entire snapshot contents with the provided endpoints, from a listing without a store revision. It returns the new version string plus the service-catalog transitions (see Apply) between the old and new contents; the caller stamps and broadcasts them. The version moves only when the contents do.

func (*Snapshot) ServiceNames 

func (s *Snapshot) ServiceNames() []string

ServiceNames returns the sorted names of all services currently holding at least one endpoint — the service catalog replayed to every new watcher.

func (*Snapshot) State 

func (s *Snapshot) State() State

State returns the snapshot's version, revision, content hash and generation, read in one critical section.

func (*Snapshot) Version 

func (s *Snapshot) Version() string

Version returns the current snapshot version (see the format above).

func (*Snapshot) WatchStart 

func (s *Snapshot) WatchStart(token string, filter, have map[string]struct{}) (events []*registrarv1.WatchEndpointsResponse, services []string, version string, resume Resume, extended bool)

WatchStart is the snapshot half of a new watch, read in one critical section. Given the client's resume token it returns the resume outcome, the unversioned FULL_SNAPSHOT events (ResumeResend only), the service catalog to replay (all but ResumeCurrent), and the current version, which the SNAPSHOT_COMPLETE marker carries in every case.

have is the request's partial_resume (#1239): nil for an ordinary token, else the services the client's cache holds at token. When the token names the current contents, the client is EXTENDED instead of resent: events are the ENDPOINT_ADDED events of filter's services outside have, and extended is true. A token that does not name the current contents is resent in full, as for an ordinary token: the client's held services may be stale too.

Extending is exact only because the content hash is over the WHOLE snapshot: a token naming the current contents means every service the client holds is current, whatever its filter was when it earned the token.

type State 

type State struct {
	// Version is the resume token (see the format above).
	Version string
	// Revision is the store revision of the last installed listing; 0 = none.
	Revision int64
	// Dirty reports that the contents deviate from the listing at Revision.
	Dirty bool
	// ContentHash names the contents; equal on two replicas iff they serve the
	// same endpoints.
	ContentHash string
	// Generation counts content changes in this process.
	Generation uint64
}

State is a consistent read of the snapshot's identity.

type Syncer 

type Syncer struct {
	// contains filtered or unexported fields
}

Syncer periodically polls an external registry, computes a diff against the local snapshot, and broadcasts changes to all watching agents. It implements the controller-runtime Runnable interface.

func NewSyncer 

func NewSyncer(reg registry.Registry, snapshot *Snapshot, broadcaster *Broadcaster, syncInterval time.Duration, log *slog.Logger, metrics *Metrics) *Syncer

NewSyncer creates a Syncer that polls the external registry at the given interval. metrics may be nil to disable instrumentation.

func (*Syncer) NeedLeaderElection 

func (s *Syncer) NeedLeaderElection() bool

NeedLeaderElection returns false so the syncer runs on all replicas, not just the leader. Each replica independently polls the external registry and broadcasts changes to its connected agents.

func (*Syncer) Start 

func (s *Syncer) Start(ctx context.Context) error

Start runs the sync loop until the context is cancelled. When the registry supports change notifications (registry.ChangeNotifier — the etcd backend's clientv3 watch, the kubernetes backend's managed-Pod informer since aether#1124), the loop syncs at watch speed (debounced) instead of only at the poll interval; the periodic poll remains a backstop for any event missed during a watch re-establish. Backends without notifications fall back to poll-only, unchanged.

func (*Syncer) Synced 

func (s *Syncer) Synced() <-chan struct{}

Synced returns a channel that is closed once the first sync cycle has completed and the snapshot reflects the external registry.

func (*Syncer) UseWriteBehind 

func (s *Syncer) UseWriteBehind(q *WriteBehindQueue)

UseWriteBehind wires the write-behind queue's pending-intent overlay into the sync cycle.

type WriteBehindQueue 

type WriteBehindQueue struct {
	// contains filtered or unexported fields
}

WriteBehindQueue flushes snapshot-first registry mutations to the external registry with retries, and shields the sync loop from un-observed intents. It implements controller-runtime's Runnable (all replicas).

func NewWriteBehindQueue 

func NewWriteBehindQueue(reg registry.Registry, log *slog.Logger, metrics *Metrics) *WriteBehindQueue

NewWriteBehindQueue creates the queue. metrics may be nil.

func (*WriteBehindQueue) EnqueueRegister 

func (q *WriteBehindQueue) EnqueueRegister(service string, protocol registryv1.Service_Protocol, ep *registryv1.ServiceEndpoint)

EnqueueRegister records a register/upsert intent. A newer op for the same (service, protocol, ip) supersedes any older one.

func (*WriteBehindQueue) EnqueueUnregister 

func (q *WriteBehindQueue) EnqueueUnregister(service string, protocol registryv1.Service_Protocol, ip string)

EnqueueUnregister records a removal intent, superseding any pending register.

func (*WriteBehindQueue) NeedLeaderElection 

func (q *WriteBehindQueue) NeedLeaderElection() bool

NeedLeaderElection returns false: each replica owns the external writes for the RPCs it received (peer replicas converge through the external registry until peer-watch lands).

func (*WriteBehindQueue) Overlay 

func (q *WriteBehindQueue) Overlay(state map[string]map[registryv1.Service_Protocol][]*registryv1.ServiceEndpoint, listedAt time.Time) int

Overlay reconciles the queue against a freshly fetched external-registry state and patches that state with still-pending intents. Called by the sync loop BEFORE Diff/Replace, so neither the broadcast events nor the snapshot regress an intent the external registry has not materialized yet.

listedAt is when the sync started listing: every intent received before it is in what the listing returned.

Release rule: a flushed op whose intent the fetched state reflects (register: key present with an equal endpoint; unregister: key absent) is done and removed. Everything else is overlaid onto the state.

On a derived backend (kubernetes) the rule is the listing's age instead: an op received before listedAt is released and the listing is taken as it is, so every replica serves the same Pod-derived endpoint (aether#1145). An op received after listedAt is overlaid as above.

It returns how many intents it overlaid: nonzero means the state is no longer exactly the store's listing, which the snapshot version must say (#1193).

func (*WriteBehindQueue) Shielding 

func (q *WriteBehindQueue) Shielding(service, ip string) bool

Shielding reports whether an intent for (service, ip) is still pending or flushed-but-unobserved. Diagnostic/test helper.

func (*WriteBehindQueue) Start 

func (q *WriteBehindQueue) Start(ctx context.Context) error

Start runs the flush loop until ctx ends. Implements Runnable.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.