Microsoft Corp. is trying to make life easier for generative artificial intelligence developers, with a host of updates to its developer tooling today.

The software and cloud giant said the updates should help teams build more capable and knowledgeable AI models, including dedicated copilots, that can fulfill a wider range of enterprise-related tasks. The announcements at Microsoft Build 2024 include some major enhancements to Microsoft Azure AI Search and the Azure OpenAI Service, plus the general availability of the Azure AI Studio platform that debuted last year.

One of the most welcome updates for generative AI developers will be the new search relevance capabilities within Microsoft Azure AI Search. That’s a data retrieval system for retrieval-augmented generation, or RAG, and enterprise search that enables AI models to leverage customer’s private data. At Build, Microsoft announced that it now supports advanced search technologies, including hybrid search and re-ranking, together with increased storage capacity and vector index size for new services, making it easier for users to scale their generative AI applications.

The company explained these enhancements will help to return more relevant search results for generative AI models, improving the accuracy of their responses. There are new data and processing integrations too, thanks to the addition of built-in image vectorization capabilities to aid native image search, plus an integration with OneLake to help connect Azure AI Search with data in Microsoft Fabric.

Azure AI Studio was introduced in preview last November, providing developers with everything they need to create a range of generative AI experiences in one place. As part of the Azure OpenAI Service, it facilitates access to an extensive collection of large language models, data integration tools for RAG, plus intelligent search capabilities, full-lifecycle model management and AI safety tools.

Now generally available, Azure AI Studio is getting some powerful new features including what the company terms “code-first development experiences. These are introduced via integrations with the Azure Developer CLI (azd) and AI Toolkit for Microsoft Visual Studio Code tools. Within the platform, users will soon be able to access the latest foundational models via a new Models-as-a-service feature, including OpenAI’s most powerful new LLM, GPT-4o.

Azure OpenAI Service itself is getting tons of new features too, including a new Assistants API to help developers create more advanced virtual assistants and chatbots with more nuanced understanding and responsiveness, the company promised.

Reference architectures and custom generative AI models

In AI development, the company announced a series of reference architectures, together with implementation guidance, to help customers design and optimize intelligent, AI-powered applications. The idea is that developer teams can simply leverage Azure patterns and practices as a kind of blueprint to quickly build private chatbots that are more reliable, cost-efficient and compliant, the company said.

The reference architectures for Azure OpenAI Service will be available soon, helping teams jumpstart chatbot development, the company said. The new landing zone accelerators are designed to standardize and automate the deployment of the cloud infrastructure required to support those applications. In addition, developers will be able to use cloud guides and service guides that provide more precise instructions for setting up the Azure services used to deliver intelligent applications.

In addition, the company announced a new model type called “Custom generative” that’s launching in preview soon. The idea is that customers start with a single document, and then the service will guide them through the schema definition and model creation process.

The main advantage is that it eliminates the need for extensive labeling of data, so users can feed their customer generative AI models with more complex documents, in a variety of formats and templates. By using LLMs to extract the relevant data fields, users will need to correct their model’s outputs only when a specific field is incorrect, Microsoft said. With this approach, models will be able to adapt as new samples are added to their training datasets, continuously improving the accuracy and relevance of their responses.

Phi-3-vision

Microsoft normally takes advantage of its close association with OpenAI to provide developers with access to the most powerful LLMs. But it also builds a few of its own, such as the Phi family of smaller LLMs, designed to support AI processing on devices such as laptops, smartphones and tablets. Available now in preview, Phi-3-vision is the latest member of that family.

It’s a new multimodal LLM that’s designed to support visual, chart, graph and table reasoning. In other words, it can understand what it’s seeing, be it a view of the world around it, images or various documents. It can transform input images and text and output responses, explaining what it sees, the company said.

For instance, users can ask questions about a chart or a specific image and Phi-3-vision will respond accurately. The model is being made available as part of Azure AI Studio’s model-as-a-service catalog, together with Phi-3-small and Phi-3-medium.

Azure AI Speech

Azure AI Speech is getting a number of new features for building higher-quality, voice-enable applications, available in preview now.

They include a new speech analytics capability that automates the end-to-end workflow for models that extract insights from audio and video data. It integrates transcription, summarization, speech recognition, speaker diarization, sentiment analysis and other capabilities. It’s designed to work with things such as customer feedback, podcasts, call center recordings, interviews and so on.

Video dubbing is also coming to Azure AI Speech. According to Microsoft, this is a new service that can translate video files into a number of supported languages, helping companies to reach global audiences with their video content. Users can create dubbing pipelines by uploading one or a series of videos and it will automatically translate that content into the selected languages.

AI safety updates

AI safety is always a major concern and Microsoft is demonstrating how seriously it takes it with its updated Azure AI Content Safety offerings.

New additions here include Custom Categories, which can be used to create custom filters for generative AI applications, so developers can filter their outputs according to their company’s responsible AI policies. Microsoft said it will enable users to develop a more precise and relevant approach to content safety, with options for both standard and rapid deployment, the latter being for any incidents that need to be addressed quickly, in under an hour.

Prompt Shields, which is available in Microsoft Azure OpenAI Service, and Groundedness Detection in AI Studio and OpenAI Service, provide additional content filtering tools for LLMs. They’re a “pivotal development” in terms of mitigating malicious prompt injection attacks, Microsoft said, where attackers aim to make manipulate generative AI models so they “hallucinate,” or generate false or inaccurate responses.

Generative AI for education

Finally, Microsoft said it’s partnering with the nonprofit educational organization Khan Academy Inc. to explore the potential of generative AI in educational settings.

The partners are planning to do a few interesting things. For instance, Microsoft is giving all U.S. K-12 educators free access to Khanmigo for Teachers, an AI-powered teaching assistant that helps teachers to free up time to engage with their students. In addition to providing free access to the service, the company is also making additional Azure resources available to ensure that it can scale to support the expected influx of new users.

Meanwhile, Khan Academy said, it’s working with Microsoft to explore how generative AI can improve math tutoring, leveraging the latest version of Phi-3, which is being trained on the nonprofit’s private educational content.

Image: SiliconANGLE/Microsoft Designer