New Feature: Total token count per orchestration invocation

[mikel-brostrom]

Ideally the total tokens should be fetch-able by an agent response callback:

    def enhanced_agent_response_callback(message: ChatMessageContent) -> None:  
        """Enhanced logging for agent responses with handoff context."""  
        if message and message.content:  
            captured_responses.append(message)  
    ...

That configures a certain orchestrator with arbitrary agent types as members (AzureAIAgent, CopilotStudioAgent, ...)

        agents, handoffs = await build_agents(args, azure_client)      
        request_orch = HandoffOrchestration[str, ChatMessageContent](        
            members=agents,        
            handoffs=handoffs,        
            agent_response_callback=enhanced_agent_response_callback,        
        )        
    
        # Convert chat history to list of messages for the orchestration    
        messages = [msg for msg in history.messages if msg.role != AuthorRole.SYSTEM]    
        messages.append(ChatMessageContent(role=AuthorRole.USER, content=message))    
        
        # Execute orchestration with the full conversation context    
        result = await request_orch.invoke(task=messages, runtime=runtime)        

This would enable orchestration evaluation on tasks solved vs tokens used 🚀

[TaoChenOSU]

Hi @mikel-brostrom,

The message received by the response callback should contain token usage of that message. In the case of streaming, the token usage will be part of the last chunk.

We have samples on how to retrieve the token usage from a message: https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/token_usage