{"revision":"c10276e953456835cdbb1f3a60d58ae1980be353","document":{"id":4657,"path":"conversational-interfaces/now-assist-in-virtual-agent/customize-severity-levels.md","title":"Customizing severity levels","bundle":null,"doc_type":null,"release":"australia","body":"\n# Customizing severity levels\n\nAssign a severity level to each failure pattern that automated evaluations can detect, so your evaluation results reflect how your organization wants to weigh each type of failure.\n\n## Available severity levels\n\nEvaluation results in Assistant Designer classify failed scenarios into failure patterns. Each pattern carries a severity level so you can tell how much impact a failure has on the conversation. From the **Testing** tab, select **Customize severity levels** to open the **Severity levels** page.\n\n\\[Omitted image \"cust-sev-levels-01.png\"\\] Alt text: Customize severity levels option on the Testing tab.\n\nThe following severity levels are available:\n\n|Severity level|Definition|\n|--------------|----------|\n|Critical|Task did not complete, or completed with wrong or harmful data.|\n|High|Didn't go well, but recoverable. Friction or incompleteness, no lasting harm.|\n|Medium|Task still succeeded. This is a polish or quality issue only.|\n\n## Default severity by failure pattern\n\nEach failure pattern that an evaluation can detect has a default severity level, grouped by the evaluation metric it belongs to. You can change the default severity level for any failure pattern to match how your organization wants to weigh it.\n\nA severity level that you set applies across your instance, not to a single evaluation. Every automated evaluation that runs after you save your changes uses the levels shown on the **Severity levels** page.\n\n\\[Omitted image \"cust-sev-levels-02.png\"\\] Alt text: Severity levels page showing the default severity for each failure pattern, grouped by metric.\n\n**Conversation success**\n\n|Failure pattern|Description|Default severity|\n|---------------|-----------|----------------|\n|Access permission errors|The assistant incorrectly informs the user that they lack permission for the request.|Critical|\n|Answered unanswerable query|The assistant provided a confident, specific answer to a query it should have flagged as unanswerable or lacking sufficient information.|Critical|\n|Incomplete information|The assistant gave a correct but incomplete response, omitting key details or steps specified in the expected information.|Critical|\n|Incomplete or incorrect input handling|The assistant failed to recognize valid user input, collect all required fields, or process the correct values before submission.|Critical|\n|Incorrect response|The assistant provided information or instructions that were incorrect or contradicted the expected information.|Critical|\n|Loop / stuck state|The assistant repeatedly asked for the same input or could not progress past a specific step, preventing task completion.|Critical|\n|Other conversation success issues|Unsuccessful conversation that doesn't match the other issue categories.|Critical|\n|Premature termination|The assistant ended the conversation without completing the requested task, such as submitting the request, providing the requested information, or performing the requested service action.|Critical|\n|Technical or service failure|The assistant encountered a technical error, system error, or service unavailability that prevented any meaningful response or task progress.|Critical|\n|Wrong action performed|The assistant performed a different action than expected, such as executing a service request instead of providing information or instructions, or performing the wrong service action.|Critical|\n\n**Conversation fluency**\n\n|Failure pattern|Description|Default severity|\n|---------------|-----------|----------------|\n|Ambiguous language|The assistant used vague or unclear language that could be misinterpreted.|Critical|\n|Contradictory language|The assistant made statements that directly contradict each other within the same response or conflict with established facts from the conversation.|Critical|\n|Grammar error|The assistant produced incorrect grammar such as subject-verb disagreement, wrong articles, incorrect prepositions, or inappropriate tense shifts.|High|\n|Inappropriate closure|The assistant closed or attempted to close the conversation at a point that is premature, contradictory, or contextually inappropriate.|Critical|\n|Internal detail exposure|The assistant exposed its system instructions, tool definitions, or internal reasoning that should not be visible to the user.|Critical|\n|Off topic or incoherent response|The assistant introduced irrelevant content or deviated from the logical flow of the conversation thread.|Critical|\n|Other fluency issue|Fluency issue that doesn't match the other issue categories.|Critical|\n|Overly verbose|The assistant provided a response that is too lengthy or wordy relative to the simplicity of the request.|High|\n|Poor formatting|The assistant used formatting unsuitable for the chat medium, such as excessive lists, broken markdown, or overwhelming picker options.|High|\n|Response repetition|The assistant repeated words, phrases, or information without a clear conversational purpose.|High|\n|Tone mismatch|The assistant was too formal, too casual, or too upbeat for the conversational context.|High|\n|Truncated or incomplete response|The assistant provided a response that ended abruptly or was missing key linguistic components needed to convey a complete thought.|Critical|\n|Unnatural or robotic phrasing|The assistant used phrasing that is stiff or awkward in a way a native speaker would not use.|High|\n\n**Faithfulness**\n\n|Failure pattern|Description|Default severity|\n|---------------|-----------|----------------|\n|Cross-document confusion|The assistant mixed up information from different reference documents, attributing details from one context to an unrelated topic.|Critical|\n|Fabricated schema structure|The assistant asked for, presented, or processed parameters in a way that contradicts the defined parameter types, choices, or structure in the schema or API contract.|Critical|\n|Fabricated skill capability|The assistant provided additional methods, capabilities, channels, catalog items, tools, or skills beyond those explicitly supported.|Critical|\n|Failed to use available info|The assistant failed to use relevant information that's explicitly available in the reference documents, tool outputs, or conversation context. This includes omitting important information, incorrectly stating that information is unavailable, prematurely ending the conversation, or asking for information that's already available.|Critical|\n|Hallucinated / fabricated information|The assistant presented facts, assumptions, conclusions, implications, or other information that is not supported by the reference documents, tool outputs, or conversation context.|Critical|\n|Incorrect contact or reference detail|The assistant provided an incorrect phone number, email address, URL, name, or other specific reference detail that contradicts the source material.|Critical|\n|Invented data after a failed lookup|The assistant invented specific data or results when a tool output returned an error, null, or empty response.|Critical|\n|Other response faithfulness issue|Faithfulness issue that doesn't match the other issue categories.|Critical|\n|Outdated information|The assistant presented information that has been superseded by more recent data available in the conversation history or tool outputs.|Critical|\n\n"}}