Snippset Feed

Need multilingual text or real-time conversations? Azure provides purpose-built translation APIs for both text and speech. The key is knowing which SDK object performs which job.

Text Translation — know the difference

Translation changes language; transliteration changes script.

client = TextTranslationClient(endpoint, credential)

# English → French
result = client.translate(
    body=["Hello world"],
    to_language=["fr"]
)
# → "Bonjour le monde"

The source language can be auto-detected or supplied with from_language. translate() can also target multiple languages in one request.

Speech Translation — remember the pipeline

Audio → TranslationRecognizer → translated text → SpeechSynthesizer → Audio

• SpeechTranslationConfig → credentials + source/target languages
• AudioConfig → microphone/file input or audio output
• TranslationRecognizer → recognizes and translates incoming speech
• SpeechSynthesizer → converts translated text back to speech

config = SpeechTranslationConfig(endpoint, credential)
config.speech_recognition_language = "en-US"
config.add_target_language("fr")

recognizer = TranslationRecognizer(config, audio_in)
result = recognizer.recognize_once()

# result.translations["fr"] → translated text

Key decision

One target + streaming/live output → event-based synthesis
Multiple target languages → manual synthesis

Event-based synthesis supports one translation target; for multiple languages, iterate through the translated results and synthesize each separately.

Remember: Translate = language, Transliterate = script, TranslationRecognizer = speech → translated text, SpeechSynthesizer = text → speech.

Azure Voice Live provides a managed speech-to-speech layer for building low-latency voice agents. Instead of manually chaining speech-to-text → LLM → text-to-speech, Voice Live exposes a real-time, bidirectional session that can listen and respond with streaming audio.

How it works

Client ⇄ WebSocket ⇄ Voice Live ⇄ Foundry Agent / Model

1. Connect to Voice Live and authenticate.

2. Configure the session — model/agent, modalities, voice, audio format and turn detection.

3. Stream microphone audio into the session.

4. Handle server events and immediately play streamed response audio.

5. Use VAD (Voice Activity Detection) to detect when the user starts/stops speaking.

# Conceptual SDK flow
connection = await connect(endpoint, credential, agent)

await connection.session.update(
    modalities=["text", "audio"],
    input_audio_format="pcm16",
    turn_detection="server_vad"
)

async for event in connection:
    if event.type == "input_audio_buffer.speech_started":
        stop_playback()          # user interrupts / barge-in

    elif event.type == "response.audio.delta":
        play(event.delta)        # streamed agent audio

Key concepts to remember

• Full-duplex / real-time: audio flows bidirectionally through a persistent connection.

• Authentication: Microsoft Entra ID is recommended; Voice Live also supports API-key authentication.

• PCM16 vs G.711: PCM16 = uncompressed/high fidelity; G.711 = smaller, speech-oriented encoding.

• input_audio_buffer.speech_started is especially important: with server VAD it signals detected user speech and can be used to stop current playback when the user interrupts.

• Voice Live can additionally provide noise suppression, echo cancellation, turn detection, voices, avatars and function calling.

Mental model: Connect → Configure Session → Stream Audio → Handle Events → React to Interruptions.

Azure Speech MCP gives a Foundry agent speech-to-text (STT) and text-to-speech (TTS) as remotely callable tools. The key idea: the agent doesn't implement speech processing itself—it selects and invokes the Speech MCP tool when needed.

Architecture

User → Foundry Agent → Speech MCP Server → Azure Speech
↕
Blob Storage

Know the responsibilities:

• Agent: interprets the request and selects the appropriate MCP tool.

• Speech MCP: exposes STT and TTS operations.

• Azure Speech: performs transcription or speech synthesis.

• Blob Storage: persists audio input/output.

Required configuration

The Speech MCP tool needs access to:

• Foundry resource → provides the Azure AI capabilities
• Model with tool calling – required so the agent can invoke MCP tools.
• Azure Storage Account + Blob Container → persists input/output audio
• X-Blob-Container-Uri → container URI including a SAS token, granting the MCP server the required Blob permissions
• Azure Speech MCP Server – added from the tool catalog and connected to the agent.

Key distinction: the model reasons about when speech is needed; the MCP tool performs the specialized speech operation.

Client-side mental model

Once the Speech MCP tool is configured on the agent, the application mainly invokes that agent:

project = AIProjectClient(endpoint, DefaultAzureCredential())
openai = project.get_openai_client()

response = openai.responses.create(
    input="Generate speech saying: Hello Azure",
    extra_body={
        "agent_reference": {
            "name": "speech-agent",
            "type": "agent_reference"
        }
    }
)

print(response.output_text) 
# Example: "Speech generated: https://.../audio.wav"

Key distinction: the client does not directly call Speech MCP here. It calls the Foundry agent, and the agent decides when to invoke its configured speech tool.

Azure AI Speech provides specialized APIs for adding speech-to-text (STT) and text-to-speech (TTS) to applications. Compared with using a general-purpose generative model for speech, dedicated Speech services provide more predictable latency, cost, audio formats, voices, logging, and output—but they do not reason about the meaning or intent of the content.

Core SDK pattern

Both directions start with SpeechConfig, which connects the SDK to the Azure/Foundry Speech resource using an endpoint + key or Entra ID credential.

Speech → Text:
SpeechConfig + input AudioConfig → SpeechRecognizer → recognize_once_async() → text + result metadata

Text → Speech:
SpeechConfig + output AudioConfig → SpeechSynthesizer → speak_text_async() → audio + result metadata

import azure.cognitiveservices.speech as speechsdk

speech = speechsdk.SpeechConfig(
    subscription=KEY, endpoint=ENDPOINT)

# Speech → Text
audio = speechsdk.audio.AudioConfig(filename="input.wav")
recognizer = speechsdk.SpeechRecognizer(speech, audio)
text = recognizer.recognize_once_async().get().text

# Text → Speech
audio = speechsdk.audio.AudioOutputConfig(filename="output.wav")
synth = speechsdk.SpeechSynthesizer(speech, audio)
synth.speak_text_async("Hello Azure!").get()

Remember the responsibilities

SpeechConfig = service connection + speech settings
AudioConfig = where audio comes from or goes
SpeechRecognizer = audio → text
SpeechSynthesizer = text → audio

For finer TTS control, use SSML (Speech Synthesis Markup Language) instead of plain text. SSML can control voice, speaking style, pronunciation, pauses, pitch, rate, and how specific values are spoken.

💡 Key distinction: Change the voice through SpeechConfig; use SSML when you need detailed control over how speech is delivered.

Voice-enabled generative AI is essentially a two-way conversion pipeline: speech → text lets an application understand spoken input, while text → speech turns generated responses back into audio. Azure AI Foundry provides specialized models for both inference tasks.

Know which model solves which problem:

A transcription model such as GPT-4o-mini-transcribe accepts audio and returns text. A TTS model such as GPT-4o-mini-tts performs the reverse and can also follow instructions affecting characteristics such as tone.

The implementation pattern is straightforward: deploy the appropriate model in Foundry → create an authenticated Azure OpenAI client → call the corresponding audio API → handle text or binary audio output. Streaming is useful for TTS because audio bytes can be consumed as they arrive rather than waiting for the complete response.

# Speech → Text
with open("speech.wav", "rb") as audio:
    text = client.audio.transcriptions.create(
        model="gpt-4o-mini-transcribe",
        file=audio
    )

# Text → Speech
with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input="Hello from Azure AI"
) as audio:
    audio.stream_to_file("speech.mp3")

Remember the direction: Transcribe = audio in, text out. TTS = text in, audio out. The audio side is binary data, so applications must correctly read input files or stream/write generated audio.

Need to identify a document’s language, extract people and places, or remove personal data? Azure Language in Foundry Tools provides purpose-built NLP capabilities without requiring an LLM.

Core capabilities to know

Key decision: prefer these specialized tools when you need predictable, focused text analysis. Compared with general-purpose LLMs, they can also be more appropriate where cost and latency matter. An LLM is better suited when the task requires broader reasoning or flexible natural-language generation.

Python mental model

from azure.identity import DefaultAzureCredential
from azure.ai.textanalytics import TextAnalyticsClient

client = TextAnalyticsClient(
    endpoint=endpoint,
    credential=DefaultAzureCredential()
)

docs = ["Alex visited Seattle last week."]

language = client.detect_language(docs)
entities = client.recognize_entities(docs)
pii = client.recognize_pii_entities(docs)

print(language[0].primary_language.name)
print([(e.text, e.category) for e in entities[0].entities])
print(pii[0].redacted_text)

Remember: endpoint + credential → TextAnalyticsClient → analysis method → structured result. Authentication can use credentials such as Microsoft Entra ID; document operations can also be performed in batches.

Scenario clues: “extract people, places and dates” → NER. “Remove customer details before publishing” → PII detection/redaction.

A single agent can handle one task well; multi-agent orchestration coordinates specialized agents to solve larger tasks together. The Microsoft Agent Framework provides orchestration patterns that can be implemented directly in application code.

Know which orchestration pattern fits which execution flow:

Key distinction: Sequential and concurrent workflows have largely predictable structures. Handoff, group chat, and especially magnetic orchestration introduce increasingly dynamic agent selection and execution.
A simple sequential workflow illustrates the core programming model:

# 1. Define specialized agents
summarizer = chat_client.as_agent(...)
classifier = chat_client.as_agent(...)
action     = chat_client.as_agent(...)

# 2. Define their execution order
workflow = SequentialBuilder(
    participants=[summarizer, classifier, action]
).build()

# 3. Run the orchestration
await workflow.run(user_input)

Here, the summarizer → classifier → action order is explicit, and each agent runs in turn.
Remember: first define the agents and their capabilities, then select an orchestration pattern matching the required execution and collaboration model.

Microsoft Agent Framework provides a code-first abstraction for building agents across models, providers, tools, conversations, and workflows. It brings together concepts from Semantic Kernel and AutoGen behind a more consistent agent programming model.

Core model to remember: Provider → Agent → Tools → Run. A provider connects the agent to an underlying model; the agent combines instructions + tools; and run() executes the interaction. Built-in capabilities include file search, web search, conversation management, and workflow orchestration.

from agent_framework import Agent, tool

@tool
def submit_claim(subject: str, body: str):
    print(subject, body)

agent = Agent(
    client=model_client,
    instructions="Create expense claims using the available tool.",
    tools=[submit_claim]
)

result = await agent.run("Submit my expenses")

The important part is what is missing: no manual function-call dispatch loop. The framework can recognize the model's tool request, invoke the registered function, return its result, and continue execution automatically.

Key concepts

• Agent abstraction → allows different model/chat providers behind a common interface.

• @tool → exposes Python functions as agent tools; metadata can be inferred instead of manually constructing tool schemas.

• Tool approval → controls whether execution requires approval.

• Agent thread → manages conversation state and stored messages.

• Authentication → prefer token-based authentication; local development may use Azure CLI credentials, while hosted applications typically require an appropriate workload identity such as managed identity.

Remember: the framework handles much of the plumbing between LLM → agent → tool → result, letting application code focus on agent behavior rather than dispatch infrastructure.

Build Agent-Driven Workflows in Microsoft Foundry

When an AI solution becomes multi-step, don't turn one agent into a logic engine. Use a workflow to combine LLM reasoning with deterministic orchestration: agents decide what something means; workflow logic decides what happens next.

Core model: Workflow = Executors (nodes) + Edges

• Agent node → invokes an agent

• Variables → carry state/data between nodes (Local.*)

• If / Switch → deterministic branching

• For-Each → process collections without duplicating nodes

• Edges → route execution between nodes

• Structured output (JSON) → turns probabilistic agent output into predictable data for downstream logic

• Human-in-the-loop → pause, clarify, approve, or escalate

Typical patterns are Sequential, Human-in-the-loop, Group chat, and Fan-out/Fan-in.

Think in workflows

A support flow could be:

Tickets → For-Each → Triage Agent → {category, confidence} → If confidence → Route

The agent classifies the ticket and returns structured JSON. The workflow then evaluates confidence: low confidence requests clarification; Billing escalates to a human; other categories continue to a Resolution Agent.

Key distinction: use an agent for semantic reasoning; use workflow nodes/edges for loops, routing, state, and business rules.

Calling it from code

The workflow can remain server-side while application code simply starts a conversation and invokes it:

project = AIProjectClient(endpoint, DefaultAzureCredential())
openai = project.get_openai_client()

conversation = openai.conversations.create()

response = openai.responses.create(
    conversation=conversation.id,
    extra_body={
        "agent_reference": {
            "name": "support-workflow",
            "type": "agent_reference"
        }
    },
    input="Process the tickets"
)

print(response.output_text)

Remember: Agent = reasoning → Structured JSON = contract → Workflow = orchestration → Edges = routing → For-Each = iteration → Human-in-loop = controlled escalation.

Building an agent is only half the job. Publishing turns a tested Foundry agent into something users can access through Microsoft 365 Copilot and Teams.

The publish pipeline

Agent Version → Publish → Agent Endpoint → Azure Bot Service → M365 / Teams

When publishing to Microsoft 365/Teams, Foundry handles several integration steps:

• validates the selected agent version and metadata

• creates/configures an Azure Bot Service for channel routing

• generates and submits the Teams app manifest

• enables the Activity protocol for message exchange

• configures authorization based on the selected publishing scope

Azure Bot Service is the bridge between Microsoft 365/Teams and the Foundry agent.

Publishing scope = access model

With organization-wide publishing, the agent becomes discoverable to users in the tenant after approval in the Microsoft 365 admin center.

Identity & versions

A published agent has a dedicated Entra agent identity and a stable endpoint. Resources accessed using that identity need the appropriate RBAC permissions.

Also distinguish saving/versioning from publishing: agent versions can evolve independently from what users currently consume.

Remember the flow:
Version → Publish → Bot Service → Activity protocol → M365/Teams

And the key mapping: Shared → RBAC; Tenant → tenant access + admin approval.