# Observability

UDiTH Portal and the render server export **metrics** and **log output** over
OpenTelemetry (OTLP). This page describes how to enable the export and documents
every metric the two components publish.

Distributed traces are not exported.

## Enabling the export

Both components read their OpenTelemetry configuration from the `OpenTelemetry`
section. Setting an endpoint is sufficient; everything else has a default. See
[Portal Configuration](/UDiTH%20Portal/Portal%20Configuration) for the full list
of keys and for how to provide them on each platform.

Portal on Linux (Docker):

```env
OpenTelemetry__Endpoint=https://otlp.example.com
OpenTelemetry__Protocol=http/protobuf
OpenTelemetry__ServiceName=udith-portal
OpenTelemetry__ResourceAttributes=deployment.environment=production
OpenTelemetry__MetricExportInterval=30000
OpenTelemetry__Headers_file=/run/secrets/otlp_authorization
```

Portal on Windows (`Settings/sharedsettings.json`):

```json
{
  "OpenTelemetry": {
    "Endpoint": "https://otlp.example.com",
    "Protocol": "http/protobuf",
    "ServiceName": "udith-portal",
    "ResourceAttributes": "deployment.environment=production",
    "MetricExportInterval": 30000,
    "Headers": "Authorization=Bearer <token>"
  }
}
```

The render server reads the same section from its own `appsettings.json`. Give
each render server its own `ServiceName`, or rely on the `WatchdogID` attribute
described below to tell them apart.

Notes:

- Setting `OpenTelemetry:Endpoint` also forwards the log output to the same
  endpoint. A separate Serilog sink is not required.
- `OpenTelemetry:Headers` usually contains a token. It can be supplied from a
  Docker secret file or from Azure Key Vault like any other setting.
- Without an endpoint, the OpenTelemetry SDK falls back to its own default
  (`localhost:4317`). Always set the endpoint explicitly.
- `MetricExportInterval` is the time in milliseconds between two exports. It also
  determines how often the values that are read on demand (disk space, hardware
  readings) are sampled.

## What is exported

| Meter                          | Component     | Contents                                          |
| ------------------------------ | ------------- | ------------------------------------------------- |
| `portal`                       | Portal        | Model access, imports, sessions, storage          |
| `watchdog`                      | Render server | Viewer instances, streams, host hardware          |
| `Microsoft.AspNetCore.Hosting` | Portal        | Standard ASP.NET Core request metrics             |
| `Microsoft.AspNetCore.*`       | Portal        | Standard ASP.NET Core routing and Kestrel metrics |
| `System.Net.Http`              | Portal        | Standard outgoing HTTP client metrics             |
| `System.Runtime`               | Portal        | Standard .NET runtime metrics (GC, threads, JIT)  |

The four standard meters are published by .NET and ASP.NET Core, not by Portal.
Their metrics are documented under
[Built-in metrics in .NET](https://learn.microsoft.com/dotnet/core/diagnostics/built-in-metrics).
The render server exports only the `watchdog` meter.

Instrument types used below:

| Type              | Meaning                                                                  |
| ----------------- | ------------------------------------------------------------------------ |
| `Counter`         | Monotonically increasing total                                            |
| `Gauge`           | Current value, written by the component when it changes                   |
| `ObservableGauge` | Current value, read by the component each time metrics are exported       |
| `Histogram`       | Distribution of recorded values                                           |

A missing unit means the value is a plain count.

## Portal metrics

### Model file access

Recorded while a client downloads model files for browser-based viewing or local
caching.

| Name                                            | Instrument | Unit | Attributes    | Description                                                        |
| ----------------------------------------------- | ---------- | ---- | ------------- | ------------------------------------------------------------------ |
| `portal.cmr.files.requested.count`              | Counter    |      | model version | Files requested                                                     |
| `portal.cmr.files.success.count`                | Counter    |      | model version | Files returned successfully                                         |
| `portal.cmr.files.notfound.count`               | Counter    |      | model version | Files requested but not found                                       |
| `portal.cmr.files.downloaded.bytes`             | Counter    | `By` | model version | Bytes returned to clients                                           |
| `portal.cmr.files.request.duration.milliseconds`| Histogram  | `ms` | model version | Time to retrieve a file. Only recorded for successful requests       |

### ZIP archive pool

A model version is stored as an archive. Portal keeps a pool of open archive
handlers per file so that concurrent requests do not have to reopen it. See
`ModelHosting` in [Portal Configuration](/UDiTH%20Portal/Portal%20Configuration).

| Name                                             | Instrument | Unit | Attributes | Description                                     |
| ------------------------------------------------ | ---------- | ---- | ---------- | ----------------------------------------------- |
| `portal.cmr.storage.zip.created.count`           | Counter    |      | `FilePath` | Archive handlers opened                          |
| `portal.cmr.storage.zip.destroyed.count`         | Counter    |      | `FilePath` | Archive handlers closed                          |
| `portal.cmr.storage.zip.available.count`         | Gauge      |      | `FilePath` | Handlers currently free in the pool              |
| `portal.cmr.storage.zip.acquire.milliseconds`    | Histogram  | `ms` | `FilePath` | Time spent waiting for a free handler            |
| `portal.cmr.files.initiate.archive.milliseconds` | Histogram  | `ms` | `FilePath` | Time to open a new archive handler               |

A rising `zip.acquire.milliseconds` together with `zip.available.count` at zero
means the pool is too small for the load; raise
`ModelHosting:ZipArchivePoolMaxSize`.

### Model import

One import run walks through the states below. A single import produces one
`recognized` measurement and then exactly one of `invalid`, `failed` or
`completed`.

| Name                                        | Instrument | Unit | Attributes    | Description                                            |
| ------------------------------------------- | ---------- | ---- | ------------- | ------------------------------------------------------ |
| `portal.cmr.model.import.recognized.count`  | Counter    |      | model version | Imports recognised and started                          |
| `portal.cmr.model.import.invalid.count`     | Counter    |      | model version | Imports rejected during pre-validation as invalid       |
| `portal.cmr.model.import.approved.count`    | Counter    |      | model version | Imports approved for processing                         |
| `portal.cmr.model.import.completed.count`   | Counter    |      | model version | Imports completed successfully                          |
| `portal.cmr.model.import.failed.count`      | Counter    |      | model version | Imports that failed                                     |

Early in an import the model is not yet identified. Those measurements carry
`model.id = 0` and an empty `model_version.uid`.

### Models, versions and sessions

| Name                                       | Instrument      | Unit | Attributes    | Description                                                           |
| ------------------------------------------ | --------------- | ---- | ------------- | --------------------------------------------------------------------- |
| `portal.cmr.model.version.count`           | Gauge           |      | model         | Number of versions per model                                           |
| `portal.cmr.model.version.deletion.count`  | Counter         |      | model version | Model versions deleted                                                 |
| `portal.cmr.model.sessions.open.count`     | Gauge           |      | model version | Currently open viewing sessions                                        |
| `portal.cmr.preload.active.count`          | Gauge           |      | model version | Preloaded viewer instances that are running but not yet claimed         |
| `portal.cmr.model.version.storage.bytes`   | ObservableGauge | `By` | model version | Disk space occupied by the model version                               |

Use `portal.cmr.model.version.count` together with
`ModelProcessing:MaxRetainedModelVersions` to see whether automatic cleanup is
keeping up.

### Server

| Name                                        | Instrument      | Unit | Attributes   | Description                                          |
| ------------------------------------------- | --------------- | ---- | ------------ | ---------------------------------------------------- |
| `portal.signaling.watchdogs.connected.count` | Gauge           |      | *none*       | Render servers currently connected to Portal          |
| `portal.disk.total.size.bytes`              | ObservableGauge | `By` | `drive.name` | Total capacity of each fixed drive                    |
| `portal.disk.available.space.bytes`         | ObservableGauge | `By` | `drive.name` | Free space on each fixed drive                        |

`portal.signaling.watchdogs.connected.count` dropping below the number of
configured render servers is the earliest sign that browser-based viewing
capacity is reduced.

## Render server metrics

Every metric in the `watchdog` meter carries the `WatchdogID` attribute, so a
single dashboard can cover all render servers.

### Viewer instances

| Name                                                  | Instrument | Unit | Attributes                | Description                                         |
| ----------------------------------------------------- | ---------- | ---- | ------------------------- | --------------------------------------------------- |
| `watchdog.viewer.start.succeeded.count`               | Counter    |      | `Model`                   | Viewer instances started successfully                |
| `watchdog.viewer.start.failed.count`                  | Counter    |      | `Model`                   | Viewer instances that failed to start or timed out   |
| `watchdog.viewer.signaling.connection.succeeded.count`| Counter    |      | `Model`                   | Viewer instances that reached Portal                 |
| `watchdog.viewer.kill.count`                          | Counter    |      | *none*                    | Viewer instances terminated by the render server     |
| `watchdog.viewer.start.duration.milliseconds`         | Histogram  | `ms` | `Model`                   | Time from launch to a usable viewer                  |
| `watchdog.viewer.active.count`                        | Gauge      |      | `Model`, `ModelVersionUid`| Viewer instances currently running                   |
| `watchdog.viewer.idle.count`                          | Gauge      |      | *none*                    | Running instances available for a new session        |
| `watchdog.viewer.launch.queue.count`                  | Gauge      |      | *none*                    | Instances waiting in the launch queue                |
| `watchdog.viewer.not.responding.count`                | Gauge      |      | *none*                    | Instances that stopped answering health checks       |

A permanently non-zero `watchdog.viewer.launch.queue.count` means the host cannot
start instances as fast as they are requested. A rising
`watchdog.viewer.not.responding.count` usually points at the graphics driver or
at memory pressure on the host.

### Streams

A stream is one browser connected to one viewer instance.

| Name                                   | Instrument | Unit | Attributes                 | Description                        |
| -------------------------------------- | ---------- | ---- | -------------------------- | ---------------------------------- |
| `watchdog.stream.started.count`        | Counter    |      | `Model`, `ModelVersionUid` | Streams that became active          |
| `watchdog.stream.stopped.count`        | Counter    |      | `Model`, `ModelVersionUid` | Streams that ended                  |
| `watchdog.stream.active.count`         | Gauge      |      | *none*                     | Streams currently active            |
| `watchdog.stream.duration.milliseconds`| Histogram  | `ms` | `Model`, `ModelVersionUid` | How long a stream lasted            |

### Connection to Portal

| Name                                                | Instrument | Unit | Attributes | Description                                                             |
| --------------------------------------------------- | ---------- | ---- | ---------- | ----------------------------------------------------------------------- |
| `watchdog.webservices.events.received.start.count`  | Counter    |      | *none*     | Start requests received from Portal                                      |
| `watchdog.webservices.events.received.stop.count`   | Counter    |      | *none*     | Stop requests received from Portal                                       |
| `watchdog.webservices.events.received.ignored.count`| Counter    |      | *none*     | Requests ignored because no matching instance existed or they arrived out of order |
| `watchdog.client.events.start.connection.count`     | Counter    |      | *none*     | Client connection attempts initiated by the render server                |
| `watchdog.hub.reconnect.count`                      | Counter    |      | *none*     | Times the connection to Portal was re-established                        |

Repeated `watchdog.hub.reconnect.count` increases indicate an unstable network
path or a reverse proxy closing idle connections; check the WebSocket timeout of
the proxy in front of Portal.

### Host hardware

| Name                                    | Instrument      | Unit  | Attributes              | Description                       |
| --------------------------------------- | --------------- | ----- | ----------------------- | --------------------------------- |
| `watchdog.cpu.load.percent`             | ObservableGauge | `%`   | `cpu.name`              | CPU load per physical CPU          |
| `watchdog.ram.used.bytes`               | ObservableGauge | `By`  | `memory.name`           | Memory in use                      |
| `watchdog.ram.available.bytes`          | ObservableGauge | `By`  | `memory.name`           | Memory available                   |
| `watchdog.gpu.utilization.percent`      | ObservableGauge | `%`   | `gpu.uuid`, `gpu.name`  | GPU utilisation                    |
| `watchdog.gpu.memory.used.megabytes`    | ObservableGauge | `MB`  | `gpu.uuid`, `gpu.name`  | GPU memory in use                  |
| `watchdog.gpu.temperature.celsius`      | ObservableGauge | `Cel` | `gpu.uuid`, `gpu.name`  | GPU temperature                    |
| `watchdog.disk.total.size.bytes`        | ObservableGauge | `By`  | `drive.name`            | Total capacity of each fixed drive |
| `watchdog.disk.available.space.bytes`   | ObservableGauge | `By`  | `drive.name`            | Free space on each fixed drive     |

Availability:

- CPU and memory metrics are only exported on **Windows** render servers, and
  only while `EnableHardwareMonitoring` is `true`.
- GPU metrics require an NVIDIA GPU and `nvidia-smi` on the host. They are not
  exported when the polling interval `NvidiaSmiCallInterval` is set to `0`.
- Disk metrics cover fixed drives only; network shares are not reported.

## Attributes

| Attribute                            | Component     | Description                                                 |
| ------------------------------------ | ------------- | ----------------------------------------------------------- |
| `model.id`                           | Portal        | Internal numeric model id. `0` when not yet known            |
| `model.name`                         | Portal        | Model name                                                   |
| `model.external_identifier`          | Portal        | External model identifier                                    |
| `model_version.uid`                  | Portal        | Model version UID. Empty when not yet known                  |
| `model_version.external_identifier`  | Portal        | External model version identifier                            |
| `FilePath`                           | Portal        | Path of the model archive an archive handler belongs to      |
| `drive.name`                         | both          | Drive or mount point, for example `C:` or `/`                |
| `WatchdogID`                         | Render server | Identifier of the render server, from its `WatchdogId` setting |
| `Model`                              | Render server | Path of the model the viewer instance was started for        |
| `ModelVersionUid`                    | Render server | Model version UID                                            |
| `cpu.name`                           | Render server | Name of the physical CPU as reported by the host             |
| `memory.name`                        | Render server | Name of the memory sensor as reported by the host            |
| `gpu.uuid`                           | Render server | GPU UUID reported by `nvidia-smi`                            |
| `gpu.name`                           | Render server | GPU model name reported by `nvidia-smi`                      |

In the tables above, **model version** stands for the five model and model
version attributes together, and **model** for `model.id`, `model.name` and
`model.external_identifier`.

Attribute values such as model names and drive letters are chosen by the
installation. Keep the number of distinct models under control, because each
combination of attribute values becomes a separate time series in the monitoring
backend.

## Log output

With `OpenTelemetry:Endpoint` set, log records are sent to the same endpoint as
the metrics, in addition to the console and file sinks. Log levels are controlled
through the `Serilog` settings, independently of the metrics configuration; see
[Portal Configuration](/UDiTH%20Portal/Portal%20Configuration).

Log records for HTTP requests and viewing sessions contain user names only when
`ShowPII` is enabled.