At Ownlayer, we empower organizations to deliver exceptional AI experiences at scale. Our collaboration with diverse clients has given us unique insights into the strategies that drive high-performing AI teams. In this blog post, we highlight key production monitoring practices that we've observed in impactful and efficient AI implementations.

Why In-production Monitoring Matters

In traditional software deployment, product and engineering teams rely on rigorous test case validation, logging, and event tracking to maintain product quality and gather valuable user behavior data. Common tools include Datadog, Posthog, PagerDuty, Amplitude, Mixpanel, and Segment.

Deploying AI features into a product is exciting for its potential, yet challenging due to the unpredictable nature of AI. These features operate in dynamic environments where data patterns, user interactions, and external factors constantly fluctuate.

To navigate this complexity, continuous monitoring is essential. It enables teams to swiftly identify and tackle issues like model drift, performance slips, and unexpected outputs. By keeping a vigilant eye on AI in production, organizations ensure their features stay reliable and effective. This proactive approach not only prevents potential failures which could disrupt business or disappoint users, but also paves the way for valuable insights. When executed skillfully, production monitoring becomes a goldmine for critical data, helping build proprietary datasets that fuel future innovations.

What to Monitor in Production

Effective in-production monitoring involves tracking various metrics and parameters to gain insights into the model's performance and behavior. While some generic metrics are broadly applicable, the real power lies in customizing your monitoring approach. This customization should be rooted in deep product understanding and institutional expertise.

System Performance Evaluators: These are the foundational metrics that form the backbone of your AI observability strategy. They provide a comprehensive view of your model's performance in production and may include metrics such as:

• Accuracy and Performance: Track how well the model's predictions align with actual outcomes. Monitor performance metrics such as F1 score, WER, BLEU, ROUGE, METEOR

• Latency and Response Time: Measure the time the model take to generate predictions. Ensure that latency remains within acceptable limits to maintain a smooth user experience.

Custom Evaluators: While standard metrics provide a foundation, deploying AI for different use cases requires evaluations tailored to the specific application and industry. This level of customization requires a fusion of deep product insight and sector-specific expertise, knowledge that often extends far beyond the scope of typical development teams. For example, one of our clients in the legal contract review space needs to monitor for confidential information. Here, what constitutes confidential information is explicitly defined and needs to be tracked meticulously.

Similarly, in a healthcare application, monitoring might include adherence to regulatory compliance and patient safety standards, which requires a deep understanding of medical terminologies and regulations. Ultimately, you must ensure the feature consistently delivers on your core value proposition.

Triggers for Precision and Efficiency

While knowing what to monitor is important, being able to trigger evaluators with precision is also crucial. In traditional software engineering, you wouldn’t test across 50 different test cases per single interaction. Proper control over when and which evaluators run is essential for:

• Cost Control: Avoid unnecessary computations by triggering evaluators only when relevant.

• Noise Reduction: Focus on meaningful insights by activating checks in context.

• Targeted Evaluation: For example, confidentiality checks trigger only for NDAs or sensitive documents, not for every contract clause.

This approach ensures your monitoring scales efficiently, providing maximum insight with minimum overhead.

Set Up Alerts

An intelligent alerting system is a key component of production monitoring which ensures critical issues never slip through the cracks. Has your pass rate dropped below 30%? Are users consistently giving thumbs down over the last hour? Are there crucial evaluators, such as PII checks, failing? You want to ensure that there are alerts in place so you can sleep well at night.

Deploy Firewall

Sometimes the stakes are high enough that you’d like to ensure that if certain evaluations fail, the response is blocked from the end user. For example, one of our clients is a mental health support bot, and it’s crucial for them to be able to block off any direct personal criticism or self-harm-related messages.

A firewall can also protect your integrated AI from “prompt injection.” For example, a fintech bot we support flags financial advice requests as unusual.

Building Labeled Data sets

Effective in-production monitoring also provides an effective way to label data points, which are essential for ongoing improvements and unlocking future possibilities such as in-house model training, fine-tuning, segment analysis, etc. This well-labeled proprietary dataset will become your single most valuable asset in the area of generative AI.

Continuous Data Labeling:

• Automated Labeling: Intelligently tag new data points based on custom criteria.

• Smart Filtering: Easily identify datasets needing human attention.

• Human-in-the-Loop: Seamlessly incorporate expert annotations when needed.

Building Proprietary Datasets: Create datasets from your production logs with rich attributes. We’ve seen clients build test suites for future releases as well as collections of corner cases for model fine-tuning. Some clients even plan to build their own in-house models using these datasets.

Watch Out For

While monitoring AI models in production, it’s important to be mindful of potential challenges and considerations:

• Latency: Ensure that the monitoring system does not introduce significant delays that could affect user experience.

• Privacy: Comply with data privacy regulations and industry standards. Should the data be anonymized, encrypted, or always kept in your environment?

• Cost: It’s important to prioritize what is worth measuring in the first place so you don’t overrun your AI bills.

The Crucial Role of Custom AI Evaluations

Fundamental models must indeed provide safe and compliant responses for all users. However, individual organizations and use cases can have various standards that cannot be implemented at a fundamental level. One analogy I like to use is that a great fundamental model is like a top graduate from the best educational institute. However, before you have that top graduate start serving your clients, you’d like to make sure that they understand your standard of excellence, and excel in it. You also need to introduce to monitor their work and measure their impact to ensure they continously improve.

In-production monitoring is vital for managing AI-integrated features, ensuring consistent and reliable performance. Embracing a robust and tailored monitoring strategy is crucial for the success and sustainability of AI initiatives in dynamic production environments.

---

Ready to elevate your AI monitoring strategy? Ownlayer offers comprehensive tools and expertise to help you implement effective in-production monitoring tailored to your specific needs. Contact us to learn more. Visit Ownlayer.com to get started.

We Talk to Customers Through Hundreds of Meetings. What else would you like to learn? Got thoughts you want to share? Comment below or find us at team@ownlayer.com