Authentication between microservices is one of those things that seems simple until you actually try to implement it.

We went through 4 different patterns over 18 months. Each one solved some problems but created new ones. Here’s what we learned, and what we eventually settled on.

The problem

You have 15 microservices. API Gateway authenticates users with JWT. But how do internal services verify requests from each other?

Service A needs to call Service B. Service B needs to know:

  1. Is this request from a legitimate service?
  2. Which user is this on behalf of?
  3. What permissions does this service/user have?

Seems straightforward. It’s not.

Pattern 1: Shared JWT secret (Don’t do this)

Our first approach: just pass the user’s JWT between services.

// API Gateway
app.post('/api/orders', authenticateUser, async (req, res) => {
  const token = req.headers.authorization;
  
  // Pass token to order service
  const order = await axios.post('http://order-service/orders', {
    userId: req.user.id,
    items: req.body.items
  }, {
    headers: { Authorization: token }
  });
  
  res.json(order.data);
});

// Order Service
app.post('/orders', verifyJWT, async (req, res) => {
  // Same JWT verification as gateway
  const order = await createOrder(req.user.id, req.body.items);
  res.json(order);
});

Why we thought it would work:

  • Simple
  • Only one JWT secret to manage
  • User context flows naturally

Why it failed:

  1. Every service needs the JWT secret - security nightmare
  2. No service identity - can’t tell if caller is Gateway or random service
  3. Token expiration - user’s token expires mid-request chain
  4. Permission confusion - user permissions ≠ service permissions

Lasted 2 months before we abandoned it.

Pattern 2: Service-to-service API keys

Next attempt: each service has an API key.

// Order Service
const SERVICE_KEYS = {
  'gateway': 'sk_gateway_abc123',
  'payment': 'sk_payment_xyz789',
  'notification': 'sk_notif_def456'
};

function authenticateService(req, res, next) {
  const apiKey = req.headers['x-api-key'];
  const service = Object.keys(SERVICE_KEYS).find(
    name => SERVICE_KEYS[name] === apiKey
  );
  
  if (!service) {
    return res.status(401).json({ error: 'Invalid API key' });
  }
  
  req.service = service;
  next();
}

app.post('/orders', authenticateService, async (req, res) => {
  // req.service tells us who called
  // req.body.userId tells us on behalf of whom
  const order = await createOrder(req.body.userId, req.body.items);
  res.json(order);
});

Better than JWT:

  • Services have identity
  • Keys don’t expire
  • Can revoke specific service’s access

Still had problems:

  1. Key management hell - 15 services × 14 potential callers = 210 key pairs
  2. Key rotation - how do you rotate without downtime?
  3. No user context - have to pass userId separately
  4. Still just strings in config - easy to leak

Lasted 4 months. Key rotation broke prod twice.

Pattern 3: Mutual TLS (mTLS)

Heard about mTLS from a conference talk. Sounded perfect.

Each service gets a certificate. Services verify each other’s certs.

# docker-compose.yml excerpt
order-service:
  volumes:
    - ./certs/order-service.crt:/certs/service.crt
    - ./certs/order-service.key:/certs/service.key
    - ./certs/ca.crt:/certs/ca.crt

// Order Service
const https = require('https');
const fs = require('fs');

const options = {
  cert: fs.readFileSync('/certs/service.crt'),
  key: fs.readFileSync('/certs/service.key'),
  ca: fs.readFileSync('/certs/ca.crt'),
  requestCert: true,
  rejectUnauthorized: true
};

https.createServer(options, app).listen(443, () => {
  console.log('Secure server running');
});

// Verify calling service
function checkServiceAuth(req, res, next) {
  const cert = req.socket.getPeerCertificate();
  if (!cert || !cert.subject) {
    return res.status(401).json({ error: 'No client certificate' });
  }
  
  req.callingService = cert.subject.CN;
  next();
}

Pros:

  • Actually secure
  • No shared secrets
  • Built into TLS protocol
  • Industry standard

Cons:

  • Certificate management is hard
  • Renewal automation is painful
  • Debugging connection issues is hell
  • Need service mesh (Istio/Linkerd) to make it bearable

We tried running our own CA. Don’t do that. Use a service mesh or don’t use mTLS.

Lasted 6 months until we migrated to Kubernetes with Istio.

Pattern 4: JWTs with service mesh (What we use now)

Current setup: Istio handles mTLS between services. We use JWTs for identity and permissions.

But not user JWTs. Service JWTs.

// JWT payload for service-to-service calls
{
  "iss": "auth-service",
  "sub": "service:order-service",
  "aud": "service:payment-service",
  "iat": 1706745600,
  "exp": 1706745900,  // 5 minutes
  "scope": "order:create payment:charge",
  "user_id": "user_123",  // Original user context
  "user_roles": ["customer"],
  "trace_id": "abc-123"  // For logging
}

How it works:

  1. API Gateway authenticates user with user JWT
  2. Gateway generates service JWT for downstream calls
  3. Services verify service JWT, not user JWT
  4. User context flows as claims, not as auth

Code:

// Shared auth library
const jwt = require('jsonwebtoken');
const SERVICE_KEY = process.env.SERVICE_JWT_SECRET;

function createServiceToken(opts) {
  return jwt.sign({
    sub: `service:${process.env.SERVICE_NAME}`,
    aud: `service:${opts.targetService}`,
    scope: opts.scope,
    user_id: opts.userId,
    user_roles: opts.userRoles,
    trace_id: opts.traceId
  }, SERVICE_KEY, {
    expiresIn: '5m',
    issuer: 'auth-service'
  });
}

function verifyServiceToken(token) {
  return jwt.verify(token, SERVICE_KEY, {
    audience: `service:${process.env.SERVICE_NAME}`,
    issuer: 'auth-service'
  });
}

// API Gateway
app.post('/api/orders', authenticateUser, async (req, res) => {
  // Create service token for order service
  const serviceToken = createServiceToken({
    targetService: 'order-service',
    scope: 'order:create',
    userId: req.user.id,
    userRoles: req.user.roles,
    traceId: req.id
  });
  
  const response = await axios.post('http://order-service/orders', {
    items: req.body.items
  }, {
    headers: { 
      'Authorization': `Bearer ${serviceToken}`,
      'X-Trace-Id': req.id
    }
  });
  
  res.json(response.data);
});

// Order Service
app.post('/orders', verifyServiceAuth, async (req, res) => {
  // req.service contains verified service token claims
  // req.service.user_id is the original user
  // req.service.scope contains permissions
  
  if (!req.service.scope.includes('order:create')) {
    return res.status(403).json({ error: 'Insufficient permissions' });
  }
  
  const order = await createOrder(req.service.user_id, req.body.items);
  
  // Need to call payment service
  const paymentToken = createServiceToken({
    targetService: 'payment-service',
    scope: 'payment:charge',
    userId: req.service.user_id,
    userRoles: req.service.user_roles,
    traceId: req.service.trace_id
  });
  
  await chargePayment(order, paymentToken);
  
  res.json(order);
});

Why this works

1. Service identity Each JWT specifies caller (sub) and target (aud). Services can verify who’s calling.

2. Short-lived tokens 5-minute expiration. Even if leaked, quickly becomes useless.

3. Scoped permissions Each call specifies what it can do. Principle of least privilege.

4. User context preserved user_id and user_roles flow through the call chain for authorization and auditing.

5. Single secret One SERVICE_JWT_SECRET shared between all internal services. Easier than 210 API keys.

6. mTLS for transport Istio handles TLS between services. JWT handles authorization.

Implementation tips

1. Separate user auth from service auth

Don’t use user JWTs for service-to-service calls. Generate new tokens.

2. Short expiration times

5 minutes max. Most calls complete in seconds.

3. Include trace IDs

Helps debug request flows across services.

{
  "trace_id": "abc-123",
  "span_id": "def-456"
}

4. Scope permissions explicitly

Don’t give services blanket access. Specify exactly what each call can do.

// Bad
scope: "admin"

// Good
scope: "order:create payment:charge"

5. Log everything

function verifyServiceAuth(req, res, next) {
  try {
    const token = extractToken(req);
    const claims = verifyServiceToken(token);
    
    logger.info('Service call authenticated', {
      caller: claims.sub,
      target: claims.aud,
      user: claims.user_id,
      scope: claims.scope,
      trace_id: claims.trace_id
    });
    
    req.service = claims;
    next();
  } catch (err) {
    logger.warn('Service auth failed', {
      error: err.message,
      token: token.substring(0, 20)
    });
    res.status(401).json({ error: 'Unauthorized' });
  }
}

What about the secret?

Yeah, all services need SERVICE_JWT_SECRET. Still a shared secret.

But:

  • It’s internal only (never leaves cluster)
  • Only signs service tokens (not user tokens)
  • Easier to rotate (all services restart with new secret)
  • Istio mTLS protects it in transit

We rotate it monthly with zero downtime:

  1. Deploy all services with both old and new secrets
  2. Services accept tokens signed with either
  3. After 5 minutes (max token lifetime), all tokens use new secret
  4. Remove old secret

Other patterns we considered

Service mesh only (no JWT)
Istio mTLS verifies service identity. But doesn’t help with user context or permissions.

OAuth2 client credentials
Overcomplicated for internal services. Adds auth server dependency.

Signed requests (AWS SigV4)
Sign each request with service credentials. Works but complex to implement.

Lessons learned

  1. Don’t use user auth for service auth - Different security models
  2. Service mesh + JWT is powerful - Mesh for transport, JWT for authorization
  3. Keep tokens short-lived - 5 minutes is plenty
  4. Log service call patterns - Helps debug and audit
  5. Start simple - You can always add complexity later

We’re happy with this setup. Been running 8 months with zero auth-related incidents.

The key insight: services are not users. They need different authentication, different permissions, different token lifetimes.

Treat them differently and life gets easier.