Skip to content

Config

Bases: ReferenceCompletionConfigSchema, ObservedConsistencyConfigSchema, SelfReflectionConfigSchema, SemanticEvalsConfigSchema, ModelProviderSchema

Configuration for TLM inference.

This class combines multiple configuration schemas to provide comprehensive control over TLM's inference behavior, including reference completions, consistency checking, self-reflection, semantic evaluation, and model provider settings.

Attributes:

Name Type Description
quality_preset QualityPreset

Quality preset controlling the trade-off between speed and accuracy.
reasoning_effort ReasoningEffort | None

Optional reasoning effort level for models that support it.
similarity_measure SimilarityMeasure | None

Optional similarity measure to use for comparing consistency across responses.
constrain_outputs list[str] | None

Optional list of allowed output values to constrain responses, for example in multiple choice questions.
Source code in tlm/config/schema.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
class Config(
    ReferenceCompletionConfigSchema,
    ObservedConsistencyConfigSchema,
    SelfReflectionConfigSchema,
    SemanticEvalsConfigSchema,
    ModelProviderSchema,
):
    """Configuration for TLM inference.

    This class combines multiple configuration schemas to provide comprehensive
    control over TLM's inference behavior, including reference completions,
    consistency checking, self-reflection, semantic evaluation, and model provider settings.

    Attributes:
        quality_preset: Quality preset controlling the trade-off between speed and accuracy.
        reasoning_effort: Optional reasoning effort level for models that support it.
        similarity_measure: Optional similarity measure to use for comparing consistency across responses.
        constrain_outputs: Optional list of allowed output values to constrain responses, for example in multiple choice questions.
    """

    quality_preset: QualityPreset = QualityPreset.MEDIUM
    reasoning_effort: ReasoningEffort | None = None
    similarity_measure: SimilarityMeasure | None = None
    constrain_outputs: list[str] | None = None

Bases: BaseModel

Configuration for reference completion generation.

Attributes:

Name Type Description
num_reference_completions int | None

The attempted number of reference completions to generate.
Source code in tlm/config/schema.py 
 7
 8
 9
10
11
12
13
14
15
16
17
class ReferenceCompletionConfigSchema(BaseModel):
    """
    Configuration for reference completion generation.

    Attributes:
        num_reference_completions: The attempted number of reference completions to generate.
    """

    num_reference_completions: int | None = Field(
        default=None, description="The attempted number of reference completions to generate."
    )

Bases: BaseModel

Configuration for generating additional completions against which to score consistency of reference completions.

Attributes:

Name Type Description
num_consistency_completions int | None

The attempted number of observed consistency completions to generate.
observed_consistency_temperature float | None

The temperature to use for generating comparison completions.
Source code in tlm/config/schema.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
class ObservedConsistencyConfigSchema(BaseModel):
    """
    Configuration for generating additional completions against which to score consistency of reference completions.

    Attributes:
        num_consistency_completions: The attempted number of observed consistency completions to generate.
        observed_consistency_temperature: The temperature to use for generating comparison completions.
    """

    num_consistency_completions: int | None = Field(
        default=None, description="The attempted number of observed consistency completions to generate."
    )
    observed_consistency_temperature: float | None = None

Bases: BaseModel

Configuration for prompting LLM-as-judge to score the trustworthiness of reference completions using self-reflection prompts.

Attributes:

Name Type Description
self_reflection_temperature float | None

The temperature to use for self reflection completions.
num_self_reflection_completions int | None

The attempted number of self reflection completions to generate.
Source code in tlm/config/schema.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
class SelfReflectionConfigSchema(BaseModel):
    """
    Configuration for prompting LLM-as-judge to score the trustworthiness of reference completions using self-reflection prompts.

    Attributes:
        self_reflection_temperature: The temperature to use for self reflection completions.
        num_self_reflection_completions: The attempted number of self reflection completions to generate.
    """

    self_reflection_temperature: float | None = None
    num_self_reflection_completions: int | None = Field(
        default=None,
        description=(
            "The number of self reflection prompts to use. Note that the first X number of prompts will be used, "
            "i.e. the order of the prompt templates in SELF_REFLECTION_TEMPLATES_BY_WORKFLOW[workflow_type] matters. "
            "-1 means all prompts will be used."
        ),
    )

Bases: BaseModel

Configuration for semantic evaluation of reference completions.

Attributes:

Name Type Description
use_prompt_evaluation bool | None

Whether to incorporate prompt evaluation scores into the final trustworthiness score.
prompt_evaluation_temperature float | None

The temperature to use for prompt evaluation completions.
semantic_evaluation_temperature float | None

The temperature to use when generating completions to score the Evals.
Source code in tlm/config/schema.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
class SemanticEvalsConfigSchema(BaseModel):
    """
    Configuration for semantic evaluation of reference completions.

    Attributes:
        use_prompt_evaluation: Whether to incorporate prompt evaluation scores into the final trustworthiness score.
        prompt_evaluation_temperature: The temperature to use for prompt evaluation completions.
        semantic_evaluation_temperature: The temperature to use when generating completions to score the Evals.
    """

    use_prompt_evaluation: bool | None = None
    prompt_evaluation_temperature: float | None = None  # TODO: rename to prompt_evaluation_temperature
    semantic_evaluation_temperature: float | None = None  # TODO: rename to semantic_evaluation_temperature

Bases: BaseModel

Configuration for the model provider in alignment with the LiteLLM API.

Attributes:

Name Type Description
provider str | None

The name of the model provider.
api_base str | None

The base URL of the model provider's API.
api_key str | None

The API key to use for the model provider.
api_version str | None

The version of the model provider's API.
Source code in tlm/config/schema.py 
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
class ModelProviderSchema(BaseModel):
    """
    Configuration for the model provider in alignment with the LiteLLM API.

    Attributes:
        provider: The name of the model provider.
        api_base: The base URL of the model provider's API.
        api_key: The API key to use for the model provider.
        api_version: The version of the model provider's API.
    """

    provider: str | None = None
    api_base: str | None = None
    api_key: str | None = None
    api_version: str | None = None