OpenTracing on 3scale


Tags: , , ,

Categories: ,


This document will walk you through how to setup Jaeger, enable OpenTracing on APIcast and Echo-api, then confirm the integration on OpenShift 4.x.


Software requirements

My tested environment

Setup procedure

Install Red Hat OpenShift Jaeger operator

  1. Open OperatorHub.


  2. Search jaeger, then click Red Hat OpenShift Jaeger.


  3. Click Install

    Jaeger Operator Installation

  4. Click Install again.

    Jaeger Operator Installation

  5. Wait until the installation complete.

    Jaeger Operator Installation

Create Jaeger

  1. Open Installed Operators page, make sure the project is your 3scale project. Then click Jaeger link.

    Jaeger Operator Installation

  2. Click Create Jaeger.

    Jaeger Operator Installation

  3. Click Create.

    Jaeger Operator Installation

  4. jaeger-all-in-one-in-memory will be created.

    Jaeger Operator Installation

  5. Search jaeger-all-in-one-in-memory route, then open the location.

    Jaeger Operator Installation

  6. Click Login with OpenShift.

    Jaeger Operator Installation

  7. Login to OpenShift as a user.

    Jaeger Operator Installation

  8. Click Allow selected permissions.

    Jaeger Operator Installation

  9. Now you will see Jaeger UI.

    Jaeger Operator Installation

  10. You can see 4 Jaeger services in the project.

    $ oc get svc|grep '\(jaeger\|NAME\)'
    NAME                                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                  AGE
    jaeger-all-in-one-inmemory-agent                ClusterIP   None           <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP      5d
    jaeger-all-in-one-inmemory-collector            ClusterIP   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   5d
    jaeger-all-in-one-inmemory-collector-headless   ClusterIP   None           <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   5d
    jaeger-all-in-one-inmemory-query                ClusterIP    <none>        443/TCP                                  5d

Configure APIcast

Reference: Configuring APIcast with OpenTracing

  1. Configure jaeger_config.json locally.

        "service_name": "apicast",
        "disabled": false,
        "sampler": {
          "type": "const",
          "param": 1
        "reporter": {
          "queueSize": 100,
          "bufferFlushInterval": 10,
          "logSpans": false,
          "localAgentHostPort": "jaeger-all-in-one-inmemory-agent:6831"
        "headers": {
          "jaegerDebugHeader": "debug-id",
          "jaegerBaggageHeader": "baggage",
          "TraceContextHeaderName": "uber-trace-id",
          "traceBaggageHeaderPrefix": "testctx-"
        "baggage_restrictions": {
            "denyBaggageOnInitializationFailure": false,
            "hostPort": "",
            "refreshInterval": 60
  2. Register jaeger_config.json as a ConfigMap

    $ oc create configmap jaeger-config --from-file=jaeger_config.json
  3. Mount the file on apicast-production

    $ oc set volume dc/apicast-production --add --overwrite --name=jaeger-config --mount-path=/opt/app-root/src/conf.d/opentracing/jaeger_config.json --sub-path=jaeger_config.json --source='{"configMap":{"name":"jaeger-config","items":[{"key":"jaeger_config.json","path":"jaeger_config.json"}]}}'
  4. Add OPENTRACING_TRACER and OPENTRACING_CONFIG environment variables in apicast-production deploymentconfig.

    $ oc set env dc/apicast OPENTRACING_TRACER=jaeger OPENTRACING_CONFIG=/opt/app-root/src/conf.d/opentracing/jaeger_config.json
  5. After deployed apicast-production, call a production base url of a product
  6. Open Jaeger UI, select apicast service then click Find Traces.

    Find APIcast tracing

  7. Traces are listed.

    APIcast tracing

Setup echo-api

  1. Install echo-api

    $ oc new-app -f --param ECHOAPI_HOST=echo-api.apps-crc.testing
  2. Configure echo-api

    $ oc set env dc/echo-api OPENTRACING_TRACER=jaeger JAEGER_AGENT_HOST=jaeger-all-in-one-inmemory-agent JAEGER_AGENT_PORT="6831"
  3. Open Jaeger UI, select echo-api service then click Find Traces. Traces are listed.

    Echo-api tracing

Configure a product

  1. Configure the echo-api route as a backend

    Configure echo-api backend

  2. Configure the the backend to a product

    Configure product

  3. Promote the configuration to the staging and production environment

Test the integration

  1. Send a request to the production public base url
  2. Confirm whether the response returned from the echo-api


    $ curl "https://api-3scale-apicast-production.apps-crc.testing:443/?user_key=b382b63a376dc1d41de338509066e2d3" -k
      "method": "GET",
      "path": "/",
      "args": "user_key=b382b63a376dc1d41de338509066e2d3",
      "body": "",
      "headers": {
        "HTTP_VERSION": "HTTP/1.1",
        "HTTP_X_REAL_IP": "",
        "HTTP_X_3SCALE_PROXY_SECRET_TOKEN": "Shared_secret_sent_from_proxy_to_API_backend_5817cd301754f6b2",
        "HTTP_USER_AGENT": "curl/7.69.1",
        "HTTP_ACCEPT": "*/*",
        "HTTP_X_FORWARDED_HOST": "api-3scale-apicast-production.apps-crc.testing, echo-api.apps-crc.testing",
        "HTTP_X_FORWARDED_PORT": "443, 80",
        "HTTP_X_FORWARDED_PROTO": "https, http",
        "HTTP_FORWARDED": "for=;host=api-3scale-apicast-production.apps-crc.testing;proto=https, for=;host=echo-api.apps-crc.testing;proto=http",
        "HTTP_X_FORWARDED_FOR": ",",
        "HTTP_UBER_TRACE_ID": "ebd6331bd080b480:b7cca0910ca55d3d:ebd6331bd080b480:1",
        "HTTP_HOST": "echo-api.apps-crc.testing"
      "uuid": "4029588a-77de-44b8-a80a-388e5506aa50"
  3. Open Jaeger UI, select apicast service then click Find Traces. Traces are listed.

    APIcast traces

  4. Click a trace that contais apicast and echo-api. APIcast and Echo-api is included in one trace.

    Trace detail