Building services around large language model APIs may feel daunting for those rooted in traditional software development. User inputs are unpredictable, the model operates as a black box, and the outputs can be ambiguous. However, this very unpredictability can inspire fresh approaches to working with these models.

If the model’s output is structured, like in JSON format, it opens up countless possibilities for integrating the API into existing workflows. You can also define and share specific functions and arguments with the model, enabling it to call your internal services when needed.

The biggest mindset shift comes in error handling. In traditional software development, errors often signal a need to halt processes. With large language models, however, you should design for graceful failure. This includes allowing the model to self-correct by analyzing errors and making subsequent calls with adjusted arguments.

Guardrails are, of course, essential for mitigating hallucinations or unwanted outputs. But flexibility and monitoring are equally crucial. Accept that the model can and will make mistakes; use those moments as opportunities to refine and improve. While many principles from traditional development still apply, embracing this adaptability can lead to more robust and innovative systems.