Log Streaming

Log streaming lets you access and process logs of a running process in real time, or retrieve a snapshot of all logs produced so far. Use streaming for long-running or background commands. Use snapshots when you only need the current output.

Every process started through the process module writes its output to a cursor-addressed log ledger. Each log frame carries a channel (stdout and stderr for commands, pty for PTY processes), the data, and a cursor you can use to resume reading from where you left off. See log frames for the full list of fields and channels.

For entrypoint log streaming, see entrypoint process. To stream logs while sending input to a running command, see execute interactive commands.

Use cases

Log streaming is used when an application needs the output of a process while the process still runs.

• Live output in a UI: build, test, or training output appears in a web application as the process writes it. Each frame carries a timestamp, so the UI shows when the output was written.
• Agent feedback: an agent reads the output of a long command as it arrives and stops the command early on an error.
• Reconnects: a client that loses its connection resumes from its last cursor. No frame is delivered twice, and no retained frame is missed.
• Server logs: the output of a background web server or worker is followed for as long as the process runs.
• Error handling: stdout and stderr arrive on separate channels, so an application handles errors apart from regular output.
• Results of code runs: code executions stream charts, results, and exceptions on their own channels as JSON. An application renders each one as it arrives.
• Binary output: base64 encoding returns the raw bytes of each frame. Output that is not valid UTF-8 text is read without loss.
• Output after exit: a named process keeps its logs until cleanup. An application reads the retained output of a finished or failed process later, page by page. Output beyond the retention cap evicts the oldest frames.

Stream logs

Stream the logs of a process. A stream replays the retained output from the beginning or from a cursor, then new output as it is written. It ends once the process exits and all output is delivered. While the process is silent, the daemon sends a keepalive every 15 seconds.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

handle = sandbox.process.start(
    'for i in {1..5}; do echo "Step $i"; echo "Error $i" >&2; sleep 1; done'
)

for frame in handle.stream_logs():
    print(f"[{frame.channel.upper()}]: {frame.data}", end="")

Stream events

Handle the events of a log stream. Each event carries a cursor, also sent as the event ID, so a client can resume after any event. A stream ends after its end event or its error event.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

handle = sandbox.process.start("for i in {1..3}; do echo \"Step $i\"; sleep 1; done")

# Only log frames are yielded; the generator returns after the eof event
for frame in handle.stream_logs():
    print(f"[{frame.channel.upper()}]: {frame.data}", end="")

# The exit result is read separately
result = handle.wait()
print(f"Exit code: {result.exit_code}, reason: {result.reason}")

Resume a stream

Resume a stream from a cursor. A stream opened with a cursor starts strictly after that position, so no frame is delivered twice. An SSE client that reconnects sends the ID of the last event it received, and the daemon uses that ID in place of the cursor. If retention evicted the position, the stream warns and continues from the oldest retained frame.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

handle = sandbox.process.start("for i in {1..5}; do echo \"Step $i\"; sleep 1; done")

# Stop early, remembering the cursor of the last frame processed
last_cursor = None
for frame in handle.stream_logs():
    print(frame.data, end="")
    last_cursor = frame.cursor
    if frame.data.startswith("Step 3"):
        break

# Resume strictly after the last frame seen
for frame in handle.stream_logs(cursor=last_cursor):
    print(frame.data, end="")

Retrieve all existing logs

Read the logs a process has produced so far. A read returns one page and does not wait for new output. Output longer than one page needs pagination.

# Logs produced up to the current point in time
page = handle.logs()
for frame in page.items:
    print(f"[{frame.channel.upper()}]: {frame.data}", end="")

Paginate logs

Read logs page by page. Pass the next cursor of a page to read the page after it. An empty page means the read has caught up with the output; a process that is still running can produce more.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

handle = sandbox.process.start("seq 1 5000")
handle.wait()

frames = []
cursor = None
while True:
    page = handle.logs(cursor=cursor, limit=500)
    frames.extend(page.items)
    if not page.items or page.next_cursor == cursor:
        break
    cursor = page.next_cursor

stdout = "".join(frame.data for frame in frames if frame.channel == "stdout")
print(f"{len(frames)} frames, {len(stdout)} bytes of stdout")

Binary output

Read binary output without loss. The default text encoding replaces bytes that are not valid UTF-8. Base64 encoding returns the raw bytes of each frame as unpadded base64url, the URL-safe alphabet without trailing padding, so decode it with a decoder that accepts that form.

import base64

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

handle = sandbox.process.start("head -c 1024 /dev/urandom")
handle.wait()

page = handle.logs(encoding="base64")
raw = b"".join(
    base64.urlsafe_b64decode(frame.data + "=" * (-len(frame.data) % 4))
    for frame in page.items
    if frame.channel == "stdout"
)
print(f"{len(raw)} bytes")

Log frames

Log retention

Control how long a process record and its logs are kept. The retention policy is set when the process starts. Independent of the policy, retained output is capped in size; output beyond the cap evicts the oldest frames. A page read from an evicted position fails with an error that names the oldest retained position; a stream warns and continues from that position.

from daytona import Daytona
from daytona.v2 import CursorExpiredError

daytona = Daytona()
sandbox = daytona.v2.create()

# Keep the logs of an unnamed process until cleanup()
handle = sandbox.process.start("seq 1 5000", keep_logs="until_cleanup")
handle.wait()

saved_cursor = "..."  # a cursor stored from an earlier read
try:
    page = handle.logs(cursor=saved_cursor)
except CursorExpiredError as e:
    # The position was evicted; restart from the oldest retained frame
    page = handle.logs(cursor=e.first_available_cursor)

handle.cleanup()