At Ownlayer, we provide 17 different kinds of evaluators covering basic search, JSON validation, and custom evaluators that leverage LLM as the judge. This article explains how to create an LLM-powered custom evaluator.

Create a criterion

Access Criteria

• Navigate to the 'Evaluators' section from the left panel.

• Click on the 'Criteria' tab

Describe Your Criterion

• Click on 'Create criterion'.

• Describe your criterion in natural language, detailing what good performance looks like. The LLM will use this description as the standard to evaluate your data.

• Click on 'Create' to save the criterion.

Create an LLM-Powered Evaluator

Access Evaluators

• Navigate to the 'Evaluators' section from the left panel.

• Click on the 'Evaluators' tab.

Create Evaluator

• Click on the 'Create evaluator' button.

• Fill out the form to create the desired evaluator. This includes:

  • Name of the evaluator.

  • Description of what the evaluator does.

  • Select the evaluator type

  • Link the criterion you created earlier.

  • Select the LLM configuration to power the evaluator.

• Click on 'Create evaluator' to save the evaluator.

Add Trigger

Add Trigger Logic

To automatically evaluate an inference, you can add detailed triggering logic.

• Select the evaluator you just created.

• Navigate to the 'Triggers' section within the evaluator's detail page.

• Click on "Add tag"

Implement Trigger in Inference Stream

• Ensure that the tags specified in the evaluators' triggers are included when you stream your inference data.

• Follow the full API documentation for detailed instructions on how to implement triggers in your inference stream.

Need Help?

If you have any questions or need assistance, feel free to contact us at team@ownlayer.com or schedule a call with us here.