Pipecat v1.12.0
Pipecat01 · ClassifiersPipecat v1.12.0 adds Classifiers for voicemail, turn-taking and the screen
Ask typed questions about a call and get a yes or no, a choice, or a score
Pipecat adds Classifiers. A classifier answers typed questions about some state: a yes or no, a choice among options, or a score on a scale, several questions at once, each with a probability.
Classifiers
PipecatClassifiers.
JevClassifier
Pipecat01 · ClassifiersJev answers each question in a few hundred milliseconds
classifier = JevClassifier(api_key=os.getenv("TYPESAFE_API_KEY"))
results = await classifier.choice(
"I'd like to book a table for, um",
{"turn": ChoiceQuestion(
instructions="is the user's turn over?",
options={
"complete": "the user finished",
"short": "a brief pause",
"long": "asked for time",
},
)},
)
results["turn"].choice # "short"The fast backend is Jev, TypeSafe's classification model. Ask whether a caller has finished their turn, and Jev answers in a few hundred milliseconds, with how sure it is. It needs a TypeSafe API key.
LLMClassifier
Pipecat01 · ClassifiersAny Pipecat LLM can answer the same questions
classifier = LLMClassifier(
llm=OpenAILLMService(model="gpt-4o-mini"))
results = await classifier.yes_no(
"Hi, you've reached Dana. Leave a message.",
{"voicemail": YesNoQuestion(
instructions="is this a voicemail greeting?")},
)
results["voicemail"].is_yes # True
results["voicemail"].probability # 0.97LLMClassifier asks the same questions of any Pipecat LLM service. All the questions about one state go out in one call and come back as one JSON object, with a ten second timeout by default.
Voicemail detection
PipecatVoicemail detection.
VoicemailDetector waits for silence
Pipecat02 · Voicemail detectionVoicemailDetector decides after one second of silence
VoicemailDetector now asks a classifier after every transcription and acts after a second of quiet. “Hi, this is Sam” opens a real answer and a voicemail greeting alike, and the silence after it tells them apart.
One processor
Pipecat02 · Voicemail detectionOne processor replaces the parallel pipeline and its own LLM
- Takes a classifier, such as Jev
- Passing an LLM still works for now
The old gate classes and the detector’s prompt attributes are removed
The detector is now one processor, in place of a parallel pipeline with its own LLM. Passing an LLM still works for now. The old gate classes are removed, so code importing them breaks.
Screen questions
PipecatScreen questions.
The screen job
Pipecat03 · Screen questionsA voice LLM can ask about the screen without seeing it
context = LLMContext(tools=screen_tools("ui"))The UI worker has a built-in screen job. The voice LLM asks it to find an element, check the screen, or click and fill, and each answer comes back as short data, never the page.
UIWorker classifier
Pipecat03 · Screen questionsSmall screen decisions skip the LLM turn
- Which element the user means
- Whether something is true on the screen
- Whether a UI event deserves a reply
BaseUIWorker and ReplyToolMixin are deprecated, removed in 2.0
Hand the UI worker a classifier such as Jev, and it decides which element the user means, or whether an event deserves a reply, without an LLM turn. Without one, its own LLM answers more slowly.
Speech and turns
PipecatSpeech and turns.
Empty user turns
Pipecat04 · Speech and turnsA cough that cuts off the bot now gets a reply
When a cough cuts the bot off and leaves no transcript, the bot now asks the user to repeat, or picks up where it left off. Noise during a pause gets no reply by default.
Pronunciation map
Pipecat04 · Speech and turnsOne pronunciation map for every TTS service that takes hints
pronounce = CartesiaTTSService.pronunciation_transform_ipa(
{"Metformin": "mɛtˈfɔɹmɪn"})
tts = CartesiaTTSService(
text_transforms=[
("*", strip_markdown),
("*", pronounce), # last, so nothing rewrites its markup
],
)ElevenLabs: eleven_flash_v2 and eleven_turbo_v2 only
Give Pipecat a map from words to IPA, and it writes the markup each TTS service expects: Cartesia, ElevenLabs, Inworld and Deepgram Aura-2. Deepgram Flux has no pronunciation markup, so those words are spoken as written.
Sentence splitting
Pipecat04 · Speech and turnsSentence splitting drops NLTK and its data downloads
- sentencex rules ship with Pipecat
- Follows the TTS language setting
Some sentence boundaries may differ from NLTK’s Punkt
Sentence splitting before TTS now runs on sentencex rules in place of NLTK, so the data downloads and tokenizer warm-up are gone. It follows the language setting, and some sentence boundaries may land differently than before.
Transport and evals
PipecatTransport and evals.
MOQTransport reconnect
Pipecat05 · Transport and evalsMOQTransport redials a dropped relay on its own
- Redials with backoff and keeps its broadcast
- A reconnect stops replaying the RTVI log
- A missing client gets 60 seconds, up from 30
Needs moq-rs 0.4.6 or later; tested against moq-relay 0.14
In client mode, the MOQ transport now redials a dropped relay with backoff, and a reconnect stops replaying the whole RTVI log. A missing client gets a minute to return, up from thirty seconds.
Eval judge
Pipecat05 · Transport and evalsEval verdicts now come from a classifier
- An LLM explains every failed verdict
- One call per bot turn in a simulation
A borderline reply can get a different verdict than before
The eval judge now decides every verdict with a classifier, and an LLM explains each failure. Borderline replies can get a different verdict than before, and simulations cost more calls, one per bot turn.
Before you upgrade
PipecatBefore you upgrade.
New requirements
Pipecat06 · Before you upgradexAI, MCP and Transformers requirements move
- xAI STT defaults to grok-voice-transcribe-2.0
- The mcp extra needs the MCP SDK 2
- local-smart-turn and moondream need transformers 5.10
xAI speech-to-text now defaults to the second-generation Grok transcription model. The MCP extra runs only on version 2 of the MCP SDK. Local smart turn and Moondream now require Transformers 5.
Fixes
Pipecat06 · Before you upgradeWord highlighting, Flows and Telnyx bugs are fixed
- TTS word highlighting on markdown-heavy replies
- Flows node changes during an interruption
- RTVI messages over Telnyx
Word highlighting stays in step on markdown-heavy replies. A Flows node change now delivers the new context and tools even when the user interrupts. And Telnyx calls now carry RTVI messages, which were silently dropped.
Upgrade notes
Pipecat06 · Before you upgradeMost deprecated APIs keep working until Pipecat 2.0
Most of what this release deprecates, from the base UI worker to passing an LLM to the eval judge, keeps working until Pipecat 2.0. The upgrade notes for each one are in the Pipecat release notes.