Image Text To Text

1. What is Image-Text-to-Text Task?

Image-Text-to-Text is a task that leverages large models to generate natural language descriptions that match the input image and corresponding textual prompts. This task combines image perception with language generation technologies and is widely applied in creative content generation, image understanding assistance, Q&A systems, and other scenarios.

2. Typical Use Cases

• Image Description Generation: Generate detailed textual descriptions for input images, such as captions for news images or social media posts.
• Visual Question Answering: Answer questions relevant to the content of input images based on accompanying textual prompts, e.g., "How are the animals distributed in the image?"
• Creative Assistance: Extend creative content generation based on images and text keywords, such as creating scripts or story backgrounds for movies or advertisements.
• Image Super-Resolution Analysis: Conduct detailed semantic analysis of complex image content through textual generation, e.g., data annotation assistance in scientific research.

3. Key Factors Affecting Inference Results

Model Selection

Different models exhibit varying capabilities in analyzing images and generating text. Choosing the appropriate model depends on the requirements of the specific task.

Parameter Adjustments

Below are the key parameters impacting text generation from images and prompts:

Temperature

• Controls the creativity level in generated text.
  • High temperature (e.g., 1.0): Leads to more varied and rich content, but may lose precision.
  • Low temperature (e.g., 0.1): Produces more precise and stable text, but less creative.
• Use Cases: Suitable for adjusting the degree of freedom and precision in text generation as needed.

Maximum Length

• Limits the length of the generated text to ensure the output meets user requirements. For example, set shorter lengths for brief summaries or use longer lengths for detailed content analysis.
• Use Cases: Controls the level of detail and conciseness in image-text-to-text descriptions.

Top-K Sampling

• Restricts the model to consider only the top K most probable candidate words during generation.
  • Small K value: Produces more formal and accurate content but may reduce creativity.
  • Large K value: Enables more diverse and imaginative text generation.
• Use Cases:
  • For accuracy-focused results: Use smaller top_k values.
  • For diversity-focused results: Use larger top_k values.
  • If unset: Disables top_k sampling.

Top-P Sampling

• Dynamically selects words based on cumulative probability rather than a fixed count.
  • High top_p value: Generates richer and more diverse content.
  • Low top_p value: Produces more accurate and focused text on primary descriptions or answers.
• Use Cases: Balances randomness and stability by setting intermediate values.

Repetition Penalty

• Reduces the likelihood of the model repeatedly generating phrases or sentences. A higher penalty prevents redundant information and improves output quality.
• Use Cases: Prevent repetitive information in descriptions of image content, ensuring smoother and more coherent text.

Image Context Bias

• Controls the degree of association between the generated text and the input image.
  • High bias: Generates results that focus heavily on the image content itself.
  • Low bias: Combines image content and textual prompts to produce more creative results.
• Use Cases: Adjust the focus on image content based on task priorities.

4. Sample Code

import requests
url = "https://xxxxxxxxxxxx.space.opencsg.com/v1/chat/completions" #endpoint url
headers = {
    'Content-Type': 'application/json'
}
data = {
    "model": "xzgan001/InternVL2_5-1B",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {"type":"image_url", "image_url":{"url":"https://opencsg.com/images/landing_boosting_models.png"}},
                {"type":"text", "text":"Describe this image."}
            ]
        }
    ],
    "stream": True,
    "temperature": 0.2,
    "max_tokens": 200,
    "top_k": 10,
    "top_p": 0.9,
    "repetition_penalty": 1
}
response = requests.post(url=url, json=data, headers=headers, stream=True)
response.raise_for_status()