Inference Parameters for Mistral 7B

Here are key inference parameters for Mistral 7B, along with explanations and suggested values:

1. Repeat Penalty:

  • Purpose: Discourages the model from repeating itself, promoting more diverse and fluent text generation.

  • Values: 1.0 (no penalty) to 2.0 or higher for stronger penalties.

  • Recommended: Experiment with values between 1.2 and 1.8 to find the optimal balance for your specific use case.

2. Randomness (Temperature):

  • Purpose: Controls the randomness of generated text, influencing creativity and unexpectedness.

  • Values: 0.0 (deterministic) to 1.0 or higher for more random outputs.

  • Recommended: Start around 0.7 and adjust based on desired level of creativity and coherence.

3. Prompt Format:

  • Purpose: Specifies how you present prompts to the model, guiding its understanding and responses.

  • Structure: Often starts with a clear instruction or question, followed by relevant context or examples.

  • Recommendations:

    • Use clear and concise language.

    • Break down complex prompts into smaller steps.

    • Provide relevant context or examples to guide the model.

    • Consider using structured prompts with placeholders for specific information.

4. Model Initialization:

  • Purpose: Specifies which pre-trained version of Mistral 7B to load, each with distinct characteristics.

  • Options:

    • mistralai/Mistral-7B-v0.1: General-purpose language model.

    • mistralai/Mistral-7B-Instruct-v0.1: Enhanced for following instructions and staying on-topic.

  • Recommendations: Choose based on your primary use case.

5. Additional Parameters:

  • max_length: Maximum length of generated text.

  • num_beams: Number of beams for beam search decoding, influencing response quality.

  • top_p: Probability threshold for text generation, controlling diversity.

  • top_k: Number of top tokens to consider at each step, affecting randomness.

  • do_sample: Whether to use sampling for text generation, leading to more diverse and creative outputs.

Best Practices:

  • Experiment with different parameter combinations to find the optimal settings for your specific tasks.

  • Carefully consider the prompt format and provide clear instructions to the model.

  • Evaluate model responses using both qualitative and quantitative measures.

  • Stay updated on the latest advancements and best practices for using Mistral 7B.

Last updated