Using embedded etcd as distributed local store.
Thursday 09 July 2026 · 31 mins read · Viewed 8 timesTable of contents 🔗
Introduction 🔗
There isn't many documentation about embedding etcd as a distributed local store, so I wanted to write a simple article about it.
There are some situations where you do not want to add more complexity to a deployment by deploying an additional remote store. Sometimes, you feel like the level of availability of your application doesn't matter and should match the availability of the store. Or, simply, you have an actual stateful application, but want to distribute it.
In previous articles, I talked about how etcd can be used to delegate the state of the application to a distributed store to achieve high availability. Today, I will present a mix of both world: a distributed stateful application with delegated state.
Quick remainder of etcd 🔗
Etcd is a highly consistent and reliable distributed key-value store. It uses the Raft consensus algorithm to ensure that the state of the store is always consistent across the cluster.
To create a cluster, replicas of etcd are deployed, and a persistent volume is attached to each replica. Each replica knows the addresses of the other replicas.
It is a very popular distributed store for stateful applications like the Kubernetes control plane.
Basic etcd cluster settings 🔗
First, let's get familiar on how's etcd is deployed. You need these parameters:
- Storage settings:
data-dir: The directory where the data is stored.
- Network settings:
listen-client-urls: The addresses used to listen for client requests. Since we are using the embedded etcd, we will usehttp://localhost:2379. We don't want to expose to external traffic.listen-peer-urls: The addresses used to listen for peer requests. This one needs to be public for the peers to connect to each other. By default, we sethttp://0.0.0.0:2380.
- Cluster settings:
name: The identifier of the etcd instance, used for clustering.advertise-client-urls: The addresses used to advertise to the rest of the cluster. By default, we sethttp://localhost:2379.initial-advertise-peer-urls: The addresses used to advertise the peer address to the rest of the cluster. This is more delicate, you should use the public address of the peer. By default, we set tohttp://localhost:2380(for single node cluster).initial-cluster: The initial cluster configuration. This is a comma separated list of peer addresses. For example,infra0=http://localhost:2380.initial-cluster-state: The initial state of the cluster. By default, we set tonew. If you need to add new members, you set the value toexisting.initial-cluster-token: The initial token of the cluster. It should be unique to avoid conflict.
Using embedded etcd as a distributed local store 🔗
Design of the application 🔗
For the sake of the examples, we'll create a service that achieve this simple use case:
"A highly available remote configuration store for users".
Users will be able to store JSON data in the store, and retrieve it.
Bootstrapping the application 🔗
I'm going to use urfav to manage the CLI. Let's handle the flags specified
above:
1package main
2
3import (
4 "context"
5 "errors"
6 "fmt"
7 "log/slog"
8 "net/url"
9 "os"
10 "os/signal"
11 "syscall"
12 "time"
13
14 "github.com/urfave/cli/v3"
15 clientv3 "go.etcd.io/etcd/client/v3"
16 "go.etcd.io/etcd/server/v3/embed"
17)
18
19var (
20 version = "dev"
21
22 etcdName string
23 etcdListenClientURLs []string
24 etcdListenPeerURLs []string
25 etcdAdvertiseClientURLs []string
26 etcdInitialAdvertisePeerURLs []string
27 etcdDataDir string
28 etcdInitialClusterState string
29 etcdInitialCluster string
30 etcdInitialClusterToken string
31)
32
33var app = &cli.Command{
34 Name: "Distributed user store.",
35 Description: `Remote HA user store.`,
36 Suggest: true,
37 Version: version,
38 Flags: []cli.Flag{
39 &cli.StringFlag{
40 Name: "etcd.name",
41 Usage: "Name of etcd instance",
42 Value: "etcd-node",
43 Destination: &etcdName,
44 Sources: cli.EnvVars(
45 "ETCD_NAME",
46 ),
47 },
48 &cli.StringSliceFlag{
49 Name: "etcd.listen-client-urls",
50 Usage: "Listen address for client traffic. You should set to localhost or 127.0.0.1.",
51 Value: []string{"http://localhost:2379"},
52 Destination: &etcdListenClientURLs,
53 Sources: cli.EnvVars(
54 "ETCD_LISTEN_CLIENT_URLS",
55 ),
56 },
57 &cli.StringSliceFlag{
58 Name: "etcd.listen-peer-urls",
59 Usage: "Listen address for peer traffic.",
60 Value: []string{"http://0.0.0.0:2380"},
61 Destination: &etcdListenPeerURLs,
62 Sources: cli.EnvVars(
63 "ETCD_LISTEN_PEER_URLS",
64 ),
65 },
66 &cli.StringSliceFlag{
67 Name: "etcd.advertise-client-urls",
68 Usage: "Advertise address for client traffic. You should set to localhost.",
69 Value: []string{"http://localhost:2379"},
70 Destination: &etcdAdvertiseClientURLs,
71 Sources: cli.EnvVars(
72 "ETCD_ADVERTISE_CLIENT_URLS",
73 ),
74 },
75 &cli.StringSliceFlag{
76 Name: "etcd.initial-advertise-peer-urls",
77 Usage: "Initial advertise address for peer traffic. You should set to http://$(HOSTNAME).$(SERVICE_NAME)",
78 Value: []string{"http://localhost:2380"},
79 Destination: &etcdInitialAdvertisePeerURLs,
80 Sources: cli.EnvVars(
81 "ETCD_INITIAL_ADVERTISE_PEER_URLS",
82 ),
83 },
84 &cli.StringFlag{
85 Name: "etcd.initial-cluster-state",
86 Usage: "Initial cluster state for etcd",
87 Value: "new",
88 Destination: &etcdInitialClusterState,
89 Sources: cli.EnvVars(
90 "ETCD_INITIAL_CLUSTER_STATE",
91 ),
92 },
93 &cli.StringFlag{
94 Name: "etcd.initial-cluster",
95 Usage: "Initial cluster for etcd",
96 Value: "",
97 Destination: &etcdInitialCluster,
98 Sources: cli.EnvVars(
99 "ETCD_INITIAL_CLUSTER",
100 ),
101 },
102 &cli.StringFlag{
103 Name: "etcd.data-dir",
104 Usage: "Path to etcd data directory",
105 Value: "data",
106 Destination: &etcdDataDir,
107 Sources: cli.EnvVars(
108 "ETCD_DATA_DIR",
109 ),
110 },
111 &cli.StringFlag{
112 Name: "etcd.initial-cluster-token",
113 Usage: "Initial cluster token for etcd",
114 Value: "default",
115 Destination: &etcdInitialClusterToken,
116 Sources: cli.EnvVars(
117 "ETCD_INITIAL_CLUSTER_TOKEN",
118 ),
119 },
120 },
121 Action: func(ctx context.Context, _ *cli.Command) error {
122 cfg := embed.NewConfig()
123 cfg.Dir = etcdDataDir
124 cfg.Name = etcdName
125 cfg.ListenClientUrls = stringSliceToURLs(etcdListenClientURLs)
126 cfg.ListenPeerUrls = stringSliceToURLs(etcdListenPeerURLs)
127 cfg.AdvertiseClientUrls = stringSliceToURLs(etcdAdvertiseClientURLs)
128 cfg.AdvertisePeerUrls = stringSliceToURLs(etcdInitialAdvertisePeerURLs)
129 cfg.ClusterState = etcdInitialClusterState
130 cfg.InitialCluster = etcdInitialCluster
131
132 // TODO: start etcd here
133
134 <-ctx.Done()
135
136 slog.Info("shutting down", "reason", ctx.Err())
137
138 return nil
139 },
140}
141
142func stringSliceToURLs(s []string) []url.URL {
143 var urls []url.URL
144 for _, s := range s {
145 u, err := url.Parse(s)
146 if err != nil {
147 panic(err)
148 }
149 urls = append(urls, *u)
150 }
151 return urls
152}
153
154func main() {
155 ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
156 defer stop()
157 if err := app.Run(ctx, os.Args); err != nil {
158 slog.Error("failed to run", "error", err.Error())
159 os.Exit(1)
160 }
161}
We can add the following code in the TODO to start etcd:
1 slog.Info("starting etcd")
2 e, err := embed.StartEtcd(cfg)
3 if err != nil {
4 return fmt.Errorf("failed to start etcd: %w", err)
5 }
6 defer e.Close()
Then, health check it before running additional commands:
1 select {
2 case <-e.Server.ReadyNotify():
3 slog.Info("etcd started")
4 case <-time.After(60 * time.Second):
5 e.Server.Stop() // trigger a shutdown
6 return errors.New("timed out waiting for etcd to start")
7 }
And at this point, we can already test etcd cluster with Docker compose.
Testing etcd cluster with Docker compose 🔗
Here's the configuration:
1services:
2 app1:
3 image: golang:1.26.4
4 working_dir: /app
5 volumes:
6 - ./:/app
7 - app1-data:/data
8 environment:
9 ETCD_NAME: 'app1'
10 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
11 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app1:2380'
12 ETCD_INITIAL_CLUSTER_STATE: 'new'
13 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
14 ETCD_DATA_DIR: '/data'
15 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
16
17 # We expose etcd on port 2379 for testing
18 ETCD_LISTEN_CLIENT_URLS: 'http://0.0.0.0:2379'
19 ETCD_ADVERTISE_CLIENT_URLS: 'http://localhost:2379'
20 ports:
21 - '2379:2379'
22 command: ['go', 'run', './main.go']
23
24 app2:
25 image: golang:1.26.4
26 working_dir: /app
27 volumes:
28 - ./:/app
29 - app2-data:/data
30 environment:
31 ETCD_NAME: 'app2'
32 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
33 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app2:2380'
34 ETCD_INITIAL_CLUSTER_STATE: 'new'
35 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
36 ETCD_DATA_DIR: '/data'
37 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
38 command: ['go', 'run', './main.go']
39
40 app3:
41 image: golang:1.26.4
42 working_dir: /app
43 volumes:
44 - ./:/app
45 - app3-data:/data
46 environment:
47 ETCD_NAME: 'app3'
48 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
49 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app3:2380'
50 ETCD_INITIAL_CLUSTER_STATE: 'new'
51 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
52 ETCD_DATA_DIR: '/data'
53 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
54 command: ['go', 'run', './main.go']
55
56volumes:
57 app1-data:
58 app2-data:
59 app3-data:
Bring up:
1docker compose up -d
Then, install etcdctl to make health checks, and run:
1# Put foo=bar in the store
2etcdctl --endpoints=http://localhost:2379 put foo bar
3# Fetch foo
4etcdctl --endpoints=http://localhost:2379 get foo
5# Check the members
6etcdctl --endpoints=http://localhost:2379 member list
Try some chaos engineering, make app1 crash:
1docker compose restart app1
2# Check the logs
3docker compose logs app1
You should be able to see that app1 is able to rejoin the cluster. You can
also expose app2 to check if the data is consistent.
At this point, you've already initialized the etcd cluster. Since we are
embedding, there is no need to actually expose the client port. To interact with
the embedded cluster, simply use go.etcd.io/etcd/client/v3 as usual.
Handling user authentication 🔗
Let's use JWT and a PEM-encoded RSA public key to handle user authentication.
1// auth/middleware.go
2package auth
3
4import (
5 "context"
6 "crypto/x509"
7 "encoding/pem"
8 "errors"
9 "fmt"
10 "net/http"
11 "strings"
12 "time"
13
14 "github.com/go-jose/go-jose/v4"
15 "github.com/go-jose/go-jose/v4/jwt"
16)
17
18type contextKey string
19
20const ClaimsContextKey contextKey = "jwt_claims"
21
22// Middleware validates JWTs against a PEM public key
23func Middleware(pemKey []byte) (func(http.Handler) http.Handler, error) {
24 block, _ := pem.Decode(pemKey)
25 if block == nil {
26 return nil, errors.New("failed to parse PEM block")
27 }
28
29 pubKey, err := x509.ParsePKIXPublicKey(block.Bytes)
30 if err != nil {
31 return nil, fmt.Errorf("failed to parse public key: %w", err)
32 }
33
34 return func(next http.Handler) http.Handler {
35 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
36 authHeader := r.Header.Get("Authorization")
37 if authHeader == "" {
38 http.Error(w, "Authorization header missing", http.StatusUnauthorized)
39 return
40 }
41
42 parts := strings.Split(authHeader, " ")
43 if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") {
44 http.Error(w, "Invalid authorization format", http.StatusUnauthorized)
45 return
46 }
47 tokenStr := parts[1]
48
49 tok, err := jwt.ParseSigned(tokenStr, []jose.SignatureAlgorithm{jose.RS256, jose.ES256})
50 if err != nil {
51 http.Error(w, "Invalid token format", http.StatusUnauthorized)
52 return
53 }
54
55 var claims jwt.Claims
56 if err := tok.Claims(pubKey, &claims); err != nil {
57 http.Error(w, "Invalid token signature or claims", http.StatusUnauthorized)
58 return
59 }
60
61 expected := jwt.Expected{
62 Time: time.Now(),
63 }
64 if err := claims.Validate(expected); err != nil {
65 http.Error(w, "Token validation failed: "+err.Error(), http.StatusUnauthorized)
66 return
67 }
68
69 ctx := context.WithValue(r.Context(), ClaimsContextKey, claims)
70 next.ServeHTTP(w, r.WithContext(ctx))
71 })
72 }, nil
73}
74
We're injecting the claims in the context. Let's add a helper function to get the claims from the context:
1func FromContext(ctx context.Context) (jwt.Claims, bool) {
2 claims, ok := ctx.Value(ClaimsContextKey).(jwt.Claims)
3 return claims, ok
4}
Setting up the HTTP handler and server 🔗
Set up the handlers:
1// store/handler.go
2package store
3
4import (
5 "embedded-etcd/auth"
6 "encoding/json"
7 "fmt"
8 "log/slog"
9 "net/http"
10 "strings"
11
12 clientv3 "go.etcd.io/etcd/client/v3"
13)
14
15type PostRequest struct {
16 Key string `json:"key"`
17 Value any `json:"value"`
18}
19
20func namespaceKey(user string, key string) string {
21 return fmt.Sprintf("/%s/%s", user, key)
22}
23
24func PostHandler(cli *clientv3.Client) http.HandlerFunc {
25 return func(w http.ResponseWriter, r *http.Request) {
26 ctx := r.Context()
27 // Fetch user from context
28 claims, ok := auth.FromContext(ctx)
29 if !ok {
30 panic("auth middleware not called")
31 }
32
33 // Parse payload from JSON body
34 var payload PostRequest
35 if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
36 http.Error(w, err.Error(), http.StatusBadRequest)
37 return
38 }
39
40 // Serialize for etcd
41 var value strings.Builder
42 if err := json.NewEncoder(&value).Encode(payload.Value); err != nil {
43 slog.Error("failed to serialize value", "error", err.Error())
44 http.Error(w, err.Error(), http.StatusInternalServerError)
45 return
46 }
47
48 // Write to etcd
49 key := namespaceKey(claims.Subject, payload.Key)
50 _, err := cli.Put(ctx, key, value.String())
51 if err != nil {
52 http.Error(w, err.Error(), http.StatusInternalServerError)
53 return
54 }
55 }
56}
57
58func GetHandler(cli *clientv3.Client) http.HandlerFunc {
59 return func(w http.ResponseWriter, r *http.Request) {
60 ctx := r.Context()
61 // Fetch user from context
62 claims, ok := auth.FromContext(ctx)
63 if !ok {
64 panic("auth middleware not called")
65 }
66
67 // Parse payload from query parameters
68 q := r.URL.Query()
69 k := q.Get("key")
70 if k == "" {
71 http.Error(w, "key is required", http.StatusBadRequest)
72 return
73 }
74
75 // Read from etcd
76 key := namespaceKey(claims.Subject, k)
77 resp, err := cli.Get(ctx, key)
78 if err != nil {
79 http.Error(w, err.Error(), http.StatusInternalServerError)
80 return
81 }
82 if len(resp.Kvs) == 0 {
83 http.Error(w, "key not found", http.StatusNotFound)
84 return
85 }
86
87 // Value is already serialized to JSON. So, return it directly.
88 w.Write(resp.Kvs[0].Value)
89 }
90}
91
Nothing special. Notice I prefer to panic if auth middleware is not called. Since it's a critical developer error, it's better to panic to notify the developer.
Set up the server in the main:
1// main.go
2var (
3 // ...
4
5 httpListenAddr string
6 publicKeyPath string
7)
8
9var app = &cli.Command{
10 // ...
11 Flags: []cli.Flag{
12 // ...
13 &cli.StringFlag{
14 Name: "http.listen-addr",
15 Usage: "Listen address for HTTP traffic.",
16 Value: ":3000",
17 Destination: &httpListenAddr,
18 Sources: cli.EnvVars(
19 "HTTP_LISTEN_ADDR",
20 ),
21 },
22 &cli.StringFlag{
23 Name: "jwt.public-key",
24 Usage: "Path to public key for JWT authentication.",
25 Required: true,
26 Destination: &publicKeyPath,
27 Sources: cli.EnvVars(
28 "JWT_PUBLIC_KEY",
29 ),
30 },
31 },
32 Action: func(ctx context.Context, _ *cli.Command) error {
33 publicKey, err := os.ReadFile(publicKeyPath)
34 if err != nil {
35 return fmt.Errorf("failed to read public key: %w", err)
36 }
37
38 // ...
39
40 // Replace the <-ctx.Done() with the following:
41 auth, err := auth.Middleware(publicKey)
42 if err != nil {
43 return fmt.Errorf("failed to create auth middleware: %w", err)
44 }
45 http.Handle("GET /data", auth(store.GetHandler(cli)))
46 http.Handle("POST /data", auth(store.PostHandler(cli)))
47
48 slog.Info("starting server", "addr", httpListenAddr)
49 if err := http.ListenAndServe(httpListenAddr, nil); err != nil &&
50 !errors.Is(err, http.ErrServerClosed) {
51 return fmt.Errorf("failed to listen and serve HTTP server: %w", err)
52 }
53
54 // ...
55 }
56}
57
58// ...
Testing the application 🔗
Update the docker-compose:
1 services:
2 app1:
3 image: golang:1.26.4
4 working_dir: /app
5 volumes:
6 - ./:/app
7 - app1-data:/data
8+ - ./keys:/keys:ro
9 environment:
10 ETCD_NAME: 'app1'
11 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
12 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app1:2380'
13 ETCD_INITIAL_CLUSTER_STATE: 'new'
14 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
15 ETCD_DATA_DIR: '/data'
16 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
17
18 # We expose etcd on port 2379 for testing
19 ETCD_LISTEN_CLIENT_URLS: 'http://0.0.0.0:2379'
20 ETCD_ADVERTISE_CLIENT_URLS: 'http://localhost:2379'
21+ HTTP_LISTEN_ADDR: ':3000'
22+ JWT_PUBLIC_KEY: '/keys/public.pem'
23 ports:
24 - '2379:2379'
25+ - '3000:3000'
26 command: ['go', 'run', './main.go']
27
28 app2:
29 image: golang:1.26.4
30 working_dir: /app
31 volumes:
32 - ./:/app
33 - app2-data:/data
34+ - ./keys:/keys:ro
35 environment:
36 ETCD_NAME: 'app2'
37 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
38 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app2:2380'
39 ETCD_INITIAL_CLUSTER_STATE: 'new'
40 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
41 ETCD_DATA_DIR: '/data'
42 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
43+ JWT_PUBLIC_KEY: '/keys/public.pem'
44 command: ['go', 'run', './main.go']
45
46 app3:
47 image: golang:1.26.4
48 working_dir: /app
49 volumes:
50 - ./:/app
51 - app3-data:/data
52+ - ./keys:/keys:ro
53 environment:
54 ETCD_NAME: 'app3'
55 ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:2380'
56 ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://app3:2380'
57 ETCD_INITIAL_CLUSTER_STATE: 'new'
58 ETCD_INITIAL_CLUSTER: 'app1=http://app1:2380,app2=http://app2:2380,app3=http://app3:2380'
59 ETCD_DATA_DIR: '/data'
60 ETCD_INITIAL_CLUSTER_TOKEN: 'default'
61+ JWT_PUBLIC_KEY: '/keys/public.pem'
62 command: ['go', 'run', './main.go']
63
64 volumes:
65 app1-data:
66 app2-data:
67 app3-data:
Generate the public key:
1mkdir -p ./keys
2openssl genrsa -out ./keys/priv.pem 2048
3openssl rsa -in ./keys/priv.pem -pubout -out ./keys/public.pem
Re-bring up:
1docker compose up -d
Test without authentication:
1curl 'http://localhost:3000/data?key=test'
2# Authorization header missing
Let's create a JWT:
1base64url() {
2 openssl base64 -A | tr '+/' '-_' | tr -d '='
3}
4
5SUBJECT="my-user"
6PRIVATE_KEY="./keys/priv.pem"
7
8header='{"alg":"RS256","typ":"JWT"}'
9exp=$(( $(date +%s) + 300 ))
10payload=$(printf '{"sub":"%s","exp":%s}' "$SUBJECT" "$exp")
11
12# Encode
13header_b64=$(printf '%s' "$header" | base64url)
14payload_b64=$(printf '%s' "$payload" | base64url)
15unsigned="${header_b64}.${payload_b64}"
16
17# Sign
18signature=$(printf '%s' "$unsigned" | openssl dgst -sha256 -sign "$PRIVATE_KEY" | base64url)
19
20token="${unsigned}.${signature}"
Now send the request with the JWT:
1curl -H "Authorization: Bearer $token" 'http://localhost:3000/data?key=test'
2# key not found
We can confirm the authentication works, now let's create a key:
1curl -H "Authorization: Bearer $token" \
2 -H "Content-Type: application/json" \
3 -d '{"key": "my-key", "value": "my-value"}' \
4 'http://localhost:3000/data'
Now let's fetch the key:
1curl -H "Authorization: Bearer $token" 'http://localhost:3000/data?key=my-key'
2# "my-value"
It works!
Let's use etcdctl to inspect the store:
1etcdctl --endpoints=http://localhost:2379 get --prefix '/my-user'
2# /my-user/my-key
3# "my-value"
Pretty cool, huh?
A small drawback 🔗
Etcd member management is not dynamic. The only way to add a new member is to
use etcdctl member add. You can also do this with code, but, what I mean, is
that there is no auto-discovery of new members.
You need to install etcdctl alongside your program to be able to manage
the embedded etcd cluster, in case of disaster.
Conclusion 🔗
Thanks to etcd, we were able to create a distributed stateful application with high availability.
There is many other stores you can embed, like NATS JetStream for example. Pretty useful when creating a lightweight package, huh?
This is not limited to Go, you can also use Infinispan for Java. But the main requirement is that the developed application must use the same programming language as the store's programming language.
Anyway, I hope this article has been useful for you.