大型語言模型如何串流回覆

Thomas Steiner
Alexandra Klepper
Alexandra Klepper
GitHub Bluesky

發布日期：2025 年 1 月 21 日

串流 LLM 回覆是由遞增且連續發出的資料組成。 串流資料在伺服器和用戶端上看起來不同。

從伺服器

如要瞭解串流回應的樣貌，我使用指令列工具 curl 提示 Gemini 說一個長笑話。請參考以下對 Gemini API 的呼叫。如果試用，請務必將網址中的 {GOOGLE_API_KEY} 替換成您的 Gemini API 金鑰。

$ curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:streamGenerateContent?alt=sse&key={GOOGLE_API_KEY}" \
      -H 'Content-Type: application/json' \
      --no-buffer \
      -d '{ "contents":[{"parts":[{"text": "Tell me a long T-rex joke, please."}]}]}'

這項要求會以事件串流格式記錄下列 (經過截斷) 輸出內容。 每行開頭都是 data:，後面接著訊息酬載。具體格式並不重要，重要的是文字區塊。

data: {
  "candidates":[{
    "content": {
      "parts": [{"text": "A T-Rex"}],
      "role": "model"
    },
    "finishReason": "STOP","index": 0,"safetyRatings": [
      {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]
  }],
  "usageMetadata": {"promptTokenCount": 11,"candidatesTokenCount": 4,"totalTokenCount": 15}
}

data: {
  "candidates": [{
    "content": {
      "parts": [{ "text": " walks into a bar and orders a drink. As he sits there, he notices a" }],
      "role": "model"
    },
    "finishReason": "STOP","index": 0,"safetyRatings": [
      {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},
      {"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]
  }],
  "usageMetadata": {"promptTokenCount": 11,"candidatesTokenCount": 21,"totalTokenCount": 32}
}
執行指令後，結果區塊會以串流形式傳入。

第一個酬載是 JSON。請仔細查看醒目顯示的candidates[0].content.parts[0].text：

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": "A T-Rex"
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "index": 0,
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability"