Instructions to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit") model = AutoModelForCausalLM.from_pretrained("Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - MLX
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- LM Studio
- vLLM
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit
- SGLang
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Pi new
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit"
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit
Run Hermes
hermes
- MLX LM
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Docker Model Runner
How to use Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit with Docker Model Runner:
docker model run hf.co/Open4bits/Seed-OSS-36B-Instruct-mlx-2Bit
Upload chat_template.jinja with huggingface_hub
Browse files- chat_template.jinja +171 -0
chat_template.jinja
ADDED
|
@@ -0,0 +1,171 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{# ----------‑‑‑ special token variables ‑‑‑---------- #}
|
| 2 |
+
{%- set bos_token = '<seed:bos>' -%}
|
| 3 |
+
{%- set eos_token = '<seed:eos>' -%}
|
| 4 |
+
{%- set pad_token = '<seed:pad>' -%}
|
| 5 |
+
{%- set toolcall_begin_token = '<seed:tool_call>' -%}
|
| 6 |
+
{%- set toolcall_end_token = '</seed:tool_call>' -%}
|
| 7 |
+
{%- set think_begin_token = '<seed:think>' -%}
|
| 8 |
+
{%- set think_end_token = '</seed:think>' -%}
|
| 9 |
+
{%- set budget_begin_token = '<seed:cot_budget_reflect>'-%}
|
| 10 |
+
{%- set budget_end_token = '</seed:cot_budget_reflect>'-%}
|
| 11 |
+
{# -------------- reflection-interval lookup -------------- #}
|
| 12 |
+
{%- if not thinking_budget is defined %}
|
| 13 |
+
{%- set thinking_budget = -1 -%}
|
| 14 |
+
{%- endif -%}
|
| 15 |
+
{%- set budget_reflections_v05 = {
|
| 16 |
+
0: 0,
|
| 17 |
+
512: 128,
|
| 18 |
+
1024: 256,
|
| 19 |
+
2048: 512,
|
| 20 |
+
4096: 512,
|
| 21 |
+
8192: 1024,
|
| 22 |
+
16384: 1024
|
| 23 |
+
} -%}
|
| 24 |
+
{# Find the first gear that is greater than or equal to the thinking_budget. #}
|
| 25 |
+
{%- set ns = namespace(interval = None) -%}
|
| 26 |
+
{%- for k, v in budget_reflections_v05 | dictsort -%}
|
| 27 |
+
{%- if ns.interval is none and thinking_budget <= k -%}
|
| 28 |
+
{%- set ns.interval = v -%}
|
| 29 |
+
{%- endif -%}
|
| 30 |
+
{%- endfor -%}
|
| 31 |
+
{# If it exceeds the maximum gear, use the value of the last gear #}
|
| 32 |
+
{%- if ns.interval is none -%}
|
| 33 |
+
{%- set ns.interval = budget_reflections_v05[16384] -%}
|
| 34 |
+
{%- endif -%}
|
| 35 |
+
{# ---------- Preprocess the system message ---------- #}
|
| 36 |
+
{%- if messages[0]["role"] == "system" %}
|
| 37 |
+
{%- set system_message = messages[0]["content"] %}
|
| 38 |
+
{%- set loop_messages = messages[1:] %}
|
| 39 |
+
{%- else %}
|
| 40 |
+
{%- set loop_messages = messages %}
|
| 41 |
+
{%- endif %}
|
| 42 |
+
{# ---------- Ensure tools exist ---------- #}
|
| 43 |
+
{%- if not tools is defined or tools is none %}
|
| 44 |
+
{%- set tools = [] %}
|
| 45 |
+
{%- endif %}
|
| 46 |
+
{# tools2doc.jinja #}
|
| 47 |
+
{%- macro py_type(t) -%}
|
| 48 |
+
{%- if t == "string" -%}str
|
| 49 |
+
{%- elif t in ("number", "integer") -%}int
|
| 50 |
+
{%- elif t == "boolean" -%}bool
|
| 51 |
+
{%- elif t == "array" -%}list
|
| 52 |
+
{%- else -%}Any{%- endif -%}
|
| 53 |
+
{%- endmacro -%}
|
| 54 |
+
{# ---------- Output the system block ---------- #}
|
| 55 |
+
{%- if system_message is defined %}
|
| 56 |
+
{{ bos_token + "system\n" + system_message }}
|
| 57 |
+
{%- else %}
|
| 58 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 59 |
+
{{ bos_token + "system\nYou are Doubao, a helpful AI assistant. You may call one or more functions to assist with the user query." }}
|
| 60 |
+
{%- endif %}
|
| 61 |
+
{%- endif %}
|
| 62 |
+
{%- if use_json_tooldef is defined and use_json_tooldef %}
|
| 63 |
+
|
| 64 |
+
{{"Tool List:\nYou are authorized to use the following tools (described in JSON Schema format). Before performing any task, you must decide how to call them based on the descriptions and parameters of these tools."}}
|
| 65 |
+
{{ tools | tojson(ensure_ascii=False) }}
|
| 66 |
+
{%- else %}
|
| 67 |
+
{%- for item in tools if item.type == "function" %}
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
Function:
|
| 71 |
+
def {{ item.function.name }}(
|
| 72 |
+
{%- for name, spec in item.function.parameters.properties.items() %}
|
| 73 |
+
{{- name }}: {{ py_type(spec.type) }}{% if not loop.last %},{% endif %}
|
| 74 |
+
{%- endfor %}):
|
| 75 |
+
"""
|
| 76 |
+
{{ item.function.description | trim }}
|
| 77 |
+
|
| 78 |
+
{# ---------- Args ---------- #}
|
| 79 |
+
{%- if item.function.parameters.properties %}
|
| 80 |
+
Args:
|
| 81 |
+
{%- for name, spec in item.function.parameters.properties.items() %}
|
| 82 |
+
|
| 83 |
+
- {{ name }} ({{ py_type(spec.type) }})
|
| 84 |
+
{%- if name in item.function.parameters.required %} [必填]{% else %} [选填]{% endif %}:
|
| 85 |
+
{{- " " ~ (spec.description or "") }}
|
| 86 |
+
{%- endfor %}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
|
| 89 |
+
{# ---------- Returns ---------- #}
|
| 90 |
+
{%- if item.function.returns is defined
|
| 91 |
+
and item.function.returns.properties is defined
|
| 92 |
+
and item.function.returns.properties %}
|
| 93 |
+
Returns:
|
| 94 |
+
{%- for name, spec in item.function.returns.properties.items() %}
|
| 95 |
+
|
| 96 |
+
- {{ name }} ({{ py_type(spec.type) }}):
|
| 97 |
+
{{- " " ~ (spec.description or "") }}
|
| 98 |
+
{%- endfor %}
|
| 99 |
+
{%- endif %}
|
| 100 |
+
|
| 101 |
+
"""
|
| 102 |
+
{%- endfor %}
|
| 103 |
+
{%- endif %}
|
| 104 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 105 |
+
|
| 106 |
+
{{"工具调用请遵循如下格式:\n<seed:tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>value_1</parameter>\n<parameter=example_parameter_2>This is the value for the second parameter\nthat can span\nmultiple lines</parameter>\n</function>\n</seed:tool_call>\n"}}
|
| 107 |
+
{%- endif %}
|
| 108 |
+
{# End the system block line #}
|
| 109 |
+
{%- if system_message is defined or tools is iterable and tools | length > 0 %}
|
| 110 |
+
{{ eos_token }}
|
| 111 |
+
{%- endif %}
|
| 112 |
+
{# ---------- Thinking Budget ---------- #}
|
| 113 |
+
{%- if thinking_budget is defined %}
|
| 114 |
+
{%- if thinking_budget == 0 %}
|
| 115 |
+
{{ bos_token+"system" }}
|
| 116 |
+
{{ "You are an intelligent assistant that can answer questions in one step without the need for reasoning and thinking, that is, your thinking budget is 0. Next, please skip the thinking process and directly start answering the user's questions." }}
|
| 117 |
+
{{ eos_token }}
|
| 118 |
+
{%- elif not thinking_budget == -1 %}
|
| 119 |
+
{{ bos_token+"system" }}
|
| 120 |
+
{{ "You are an intelligent assistant with reflective ability. In the process of thinking and reasoning, you need to strictly follow the thinking budget, which is "}}{{thinking_budget}}{{". That is, you need to complete your thinking within "}}{{thinking_budget}}{{" tokens and start answering the user's questions. You will reflect on your thinking process every "}}{{ns.interval}}{{" tokens, stating how many tokens have been used and how many are left."}}
|
| 121 |
+
{{ eos_token }}
|
| 122 |
+
{%- endif %}
|
| 123 |
+
{%- endif %}
|
| 124 |
+
{# ---------- List the historical messages one by one ---------- #}
|
| 125 |
+
{%- for message in loop_messages %}
|
| 126 |
+
{%- if message.role == "assistant"
|
| 127 |
+
and message.tool_calls is defined
|
| 128 |
+
and message.tool_calls is iterable
|
| 129 |
+
and message.tool_calls | length > 0 %}
|
| 130 |
+
{{ bos_token + message.role }}
|
| 131 |
+
{%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
|
| 132 |
+
{{ "\n" + think_begin_token + message.reasoning_content | trim + think_end_token }}
|
| 133 |
+
{%- endif %}
|
| 134 |
+
{%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
|
| 135 |
+
{{ "\n" + message.content | trim + "\n" }}
|
| 136 |
+
{%- endif %}
|
| 137 |
+
{%- for tool_call in message.tool_calls %}
|
| 138 |
+
{%- if tool_call.function is defined %}{% set tool_call = tool_call.function %}{% endif %}
|
| 139 |
+
{{ "\n" + toolcall_begin_token + "\n<function=" + tool_call.name + ">\n" }}
|
| 140 |
+
{%- if tool_call.arguments is defined %}
|
| 141 |
+
{%- for arg_name, arg_value in tool_call.arguments | items %}
|
| 142 |
+
{{ "<parameter=" + arg_name + ">" }}
|
| 143 |
+
{%- set arg_value = arg_value if arg_value is string else arg_value | string %}
|
| 144 |
+
{{ arg_value+"</parameter>\n" }}
|
| 145 |
+
{%- endfor %}
|
| 146 |
+
{%- endif %}
|
| 147 |
+
{{ "</function>\n" + toolcall_end_token }}
|
| 148 |
+
{%- endfor %}
|
| 149 |
+
{{ eos_token }}
|
| 150 |
+
{%- elif message.role in ["user", "system"] %}
|
| 151 |
+
{{ bos_token + message.role + "\n" + message.content + eos_token }}
|
| 152 |
+
{%- elif message.role == "assistant" %}
|
| 153 |
+
{{ bos_token + message.role }}
|
| 154 |
+
{%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
|
| 155 |
+
{{ "\n" + think_begin_token + message.reasoning_content | trim + think_end_token }}
|
| 156 |
+
{%- endif %}
|
| 157 |
+
{%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
|
| 158 |
+
{{ "\n" + message.content | trim + eos_token }}
|
| 159 |
+
{%- endif %}
|
| 160 |
+
{# Include the tool role #}
|
| 161 |
+
{%- else %}
|
| 162 |
+
{{ bos_token + message.role + "\n" + message.content + eos_token }}
|
| 163 |
+
{%- endif %}
|
| 164 |
+
{%- endfor %}
|
| 165 |
+
{# ---------- Control the model to start continuation ---------- #}
|
| 166 |
+
{%- if add_generation_prompt %}
|
| 167 |
+
{{ bos_token+"assistant\n" }}
|
| 168 |
+
{%- if thinking_budget == 0 %}
|
| 169 |
+
{{ think_begin_token + "\n" + budget_begin_token + "The current thinking budget is 0, so I will directly start answering the question." + budget_end_token + "\n" + think_end_token }}
|
| 170 |
+
{%- endif %}
|
| 171 |
+
{%- endif %}
|