Logs and metrics
Everything your units write to standard output and standard error is stored, and CPU, memory and network are sampled. Both are kept 30 days, including for units that have since been scaled away and for Workloads that are stopped.
Logs
- All units merged, ordered by time, or one unit (live or gone) with
unit_id. - Each line carries its unit's slot number, the container name and the incarnation (which pod it came from), so you can tell a restart from a different copy.
- Search is a case-insensitive substring of up to 200 characters. It is not a regular expression.
- A page is 500 lines by default, 2000 at most.
- Between the lines are markers:
pod_started,pod_replaced,pod_stopped,pod_lost,pod_torn_down,container_restarted(with the reason, for exampleOOMKilled) andoverflow.
Reading backward and following
direction=backward (the default) returns the newest lines first and is how you open a log and scroll back. Hand the next_cursor back to get the next older page. direction=forward with the last follow_cursor returns what was written since; wait poll_after_ms between polls instead of hard-coding an interval. The guide Read a Workload's logs has runnable code for both.
The overflow archive
Up to 500 lines per second per container (measured over 10 seconds) go to the log store and are searchable. Nothing is dropped above that rate. The extra lines are compressed and moved to an archive kept the same 30 days, and the log shows a marker where it happened:
Code
The marker carries the time range. The console can load that range from the marker. The log download route merges the archived lines back in order, so a download is complete. The overflow count is also a chart in the metrics, log_lines_archived, so a runaway logger is easy to spot.
Metrics
GET .../units/metrics returns, for 1 hour, 6 hours, 24 hours, 7 days or 30 days, 120 evenly spaced points per series:
| window | bucket |
|---|---|
| 1h | 30 seconds |
| 6h | 3 minutes |
| 24h | 12 minutes |
| 7d | 84 minutes |
| 30d | 6 hours |
Series are cpu (millicores), memory (bytes), network_in and network_out (bytes per second) and log_lines_archived. You get the whole Workload summed and each unit's own line (at most ten, lowest slot first); omitted_units says how many were left out. Each unit's line has a limit, the ceiling it is measured against. A new Workload only offers the windows it has history for (available_windows).
Every response has a coverage sentence that says what the read actually covered, for example when part of the data was unavailable. Show it next to the chart.
The count of units
GET .../units/count charts how many units ran in the same windows: the most and fewest running in each bucket, the most billed as extra, the count the platform wanted, and markers for scaling changes. It is built from the same records as the bill.
Why a unit ended is in GET .../units/{unit_id}/terminations.