Prompt the handed model to begin a structured conversation.

[apiltamang]

Question

I'm having this issue which I think leads to a very detrimental end-user experiences. Just happened to two potential clients who were demoing interactions with an agent system based on the OpenAI Realtime API.

There is a 'customer_service_agent' who hands off to a 'delivery form agent', the latter of who specializes in taking a delivery via a structured form-like conversation. The interaction right around the handoff goes like so:

Agent CustomerServiceAgent started.

10:55:35
role='user' transcript='Yeah, I'd like to get more details from someone.'

10:55:35
Agent handoff from: CustomerServiceAgent to: DeliveryFormAgent

10:55:35
message='type='function_call' status='completed' name='transfer_to_deliveryformagent' arguments={} output=Handed off to DeliveryFormAgent'

10:55:35
Agent DeliveryFormAgent ended.

10:55:35
Agent DeliveryFormAgent started.

10:55:37
role='assistant' transcript='I’ll connect you with someone who can assist further. Thanks for reaching out!'

10:55:37
Agent DeliveryFormAgent ended.

I attempted to circumnavigate this by doing so in the backend:

                    elif event.type == "handoff":
                        logging.info(f"Agent handoff from: {event.from_agent.name} to: {event.to_agent.name}")
                        self._persist_conversation_line(f"Agent handoff from: {event.from_agent.name} to: {event.to_agent.name}")
                        self.session.send_message("Greet the caller and ask to begin form collection process.")

I don't think the last statement of getting the forms-agent to instigate its own task works (or at least reliably). Most of the item, the handoff has happened but there's just an audio that says 'I’ll connect you with someone who can assist further. Thanks for reaching out' and then there is silence.

If the caller prompts that he/she would like to proceed, of course, the forms agent does the intended task. But new users are not going to know about this, and they often wind up hanging up because the agent has gone silent.

I don't know if there's a good workaround for this. Feel like there should be an easy workaround, but still shooting in the dark.

[seratch]

I'm not sure what your actual code looks like, but you might want to provide more detailed guidance on agents' instructions, handoff name and description, tool metadata (such as name, description, parameter metadata, etc.), and anything else that might confuse agents or their tools in certain situations.

Additionally, if delivery form operations could be part of customer support communications, using the form agent as a tool could be an option as well. Handoffs often require explicit and non-overlapping instructions and metadata, so ensuring that many handoffs work together consistently in every scenario could be challenging.

[apiltamang]

This is the context for the first customer_service_agent: https://justpaste.it/ilsdk
This is the context for the second delivery_forms_agent: https://justpaste.it/aaosw

Here is the construction of the system. It's very simple with just two agents:

DeliveryFormAgent = RealtimeAgent(
    name="DeliveryFormAgent",
    instructions=open(f"{dir_path}/whr_delivery_form_context.txt").read(),
    tools=[capture_delivery_form],
    handoff_description="Handoff to this agent if customer wants to Request a delivery, pickup, or Junk Removal",
    handoffs=[]

)

CustomerServiceAgent = RealtimeAgent(
    name="CustomerServiceAgent",
    instructions=open(f"{dir_path}/whr_customer_service_context.txt").read(),
    tools=[],
    handoffs=[DeliveryFormAgent]
)

The rest is kinda boilerplate.. initialize agent, prefetch some data etc. which happens in the context of a asyncio runner bootstrapped to a VoIP call context. But the salient parts are these

    def _setup_agent_session(self, parent_ctx):
        """Run the realtime-agent session in its own asyncio loop."""
        with propagate.attach_context(parent_ctx):

            asyncio.set_event_loop(self._agent_loop)
            session_ctx = self._agent_loop.run_until_complete(self.runner.run())
            self.session = self._agent_loop.run_until_complete(session_ctx.__aenter__())

            # start background task to harvest events
            self._agent_loop.create_task(self._process_agent_events(parent_ctx))
            # Send an initial greeting to the caller as soon as the session is ready
            asyncio.run_coroutine_threadsafe(
                self.session.send_message("Greet the caller and ask how you can help!"),
                self._agent_loop
            )
            self._agent_loop.run_forever()

Note that this command: "self.session.send_message("Greet the caller and ask how you can help!")" actually works in that the assistant initiates the conversation. If I look at the history of the RealtimeItem created, this action is an entry, like so:

0: "[
  {
    \"item_id\": \"item_C3YMHHxKffETpQCnByVpe\",
    \"previous_item_id\": \"\",
    \"type\": \"message\",
    \"role\": \"user\",
    \"content\": [
      {
        \"type\": \"input_text\",
        \"text\": \"Greet the caller and ask how you can help!\",
        \"id\": null,
        \"audio\": null,
        \"transcript\": null
      }
    ],
    \"status\": \"in_progress\"
  },
  {
    \"item_id\": \"item_C3YMHFWI9Yo5IsOajode7\",
    \"previous_item_id\": null,
    \"type\": \"message\",
    \"role\": \"assistant\",
    \"status\": \"in_progress\",
    \"content\": [
      {
        \"type\": \"audio\",
        \"audio\": null,
        \"transcript\": \"Hi there! Thanks for calling We Haul Raleigh. How can I help you today?\",
        \"id\": null,
        \"text\": null
      }
    ]
  }
]"
,

My RealtimeSession handler is like so:

                async for event in self.session:
                    if event.type == "audio":
                        self.output_queue.put(event.audio.data)
                    elif event.type == "audio_end":
                        self.output_queue.put(None)
                    elif event.type == "agent_start":
                        name = getattr(event.agent, "name", None)
                        if name != getattr(self, "_last_logged_agent_name", None):
                            logging.info(f"Agent started. Name: {name}")
                            self._persist_conversation_line(f"Agent {name} started.")
                            self._last_logged_agent_name = name
                    elif event.type == "agent_end":
                        name = getattr(event.agent, "name", None)
                        if name != getattr(self, "_last_logged_agent_name", None):
                            logging.info(f"Agent ended: Name: {self._last_logged_agent_name}")
                            self._persist_conversation_line(f"Agent {name} ended.")
                            self._last_logged_agent_name = name
                    elif event.type == "handoff":
                        logging.info(f"Agent handoff from: {event.from_agent.name} to: {event.to_agent.name}")
                        self._persist_conversation_line(f"Agent handoff from: {event.from_agent.name} to: {event.to_agent.name}")
                        self.session.send_message("Greet the caller and ask to begin form collection process.")

My consternation is that this time around, "self.session.send_message("Greet the caller and ask to begin form collection process.")" does not have the same effect of causing the second handed-off agent to start the conversation. The events stack look like so:

  {
    \"item_id\": \"item_C3YN2d7Cr9wJKIqKeur8l\",
    \"previous_item_id\": \"item_C3YMi40o9LWGbSiIdEZlI\",
    \"type\": \"message\",
    \"role\": \"user\",
    \"content\": [
      {
        \"type\": \"input_audio\",
        \"audio\": null,
        \"transcript\": \"Yes, let's schedule a pickup.\",
        \"id\": null,
        \"text\": null
      }
    ],
    \"status\": \"completed\"
  },
  {
    \"item_id\": \"item_C3YN5cgR0IU5eesLMYzPa\",
    \"previous_item_id\": null,
    \"call_id\": \"call_3lkgkkVvFzKEnvxI\",
    \"type\": \"function_call\",
    \"status\": \"completed\",
    \"arguments\": \"{}\",
    \"name\": \"transfer_to_deliveryformagent\",
    \"output\": \"Handed off to DeliveryFormAgent\"
  },
  {
    \"item_id\": \"item_C3YN55Dn4vu7px2ezp7vs\",
    \"previous_item_id\": null,
    \"type\": \"message\",
    \"role\": \"assistant\",
    \"status\": \"in_progress\",
    \"content\": [
      {
        \"type\": \"audio\",
        \"audio\": null,
        \"transcript\": \"I see the phone number you're calling from is 919-800-8767. Can we use that number for this request?\",
        \"id\": null,
        \"text\": null
      }
    ]
  },
  {
    \"item_id\": \"item_C3YNJstCLckgu8CeZ5Cg1\",
    \"previous_item_id\": \"item_C3YN55Dn4vu7px2ezp7vs\",
    \"type\": \"message\",
    \"role\": \"user\",
    \"content\": [
      {
        \"type\": \"input_audio\",
        \"audio\": null,
        \"transcript\": \"Sure.\",
        \"id\": null,
        \"text\": null
      }
    ],
    \"status\": \"completed\"
  },

So two things:

1. The second time that I called **self.session.send_message("Greet the caller and ask to begin form collection process.") I don't see the a user/system call context placed in the history like i do for the first time. I was hoping there was a way to place this call context which could trigger/prompt the second agent to reliably begin its task.

2. In the context history I shared, you can see the second agent actually started on its task on handoff, which is good !! However, this behavior didn't happen earlier in the day. Also, I was hoping to eventually roll all this out on the gpt-4o-mini-realtime rarely exhibits same behavior (i.e. self start the conversation). I usually have to prompt it on the call to get it started on data collection. I was hoping a programmed prompt like "self.session.send_message(...)" would've done the trick.

A lot.. i understand. Any feedback is appreciated, although I assume there may not be something that works deterministically.

[apiltamang]

I managed to resolve the issue. I had to use

                 await self.session.send_message("Greet the caller and ask to begin form collection process.")

I had a hunch, consulted with GPT-5 (thinking), and it gave me a solid answer. ChatGPT's been terrific at solving such bugs at me. Never much been a vibe coder, to be honest!