什么时候能支持Qwen3.8-27B,有day 0 镜像吗

请问什么时候能支持Qwen3.8-27B,有day 0 镜像吗，新的0.25.0版本支持Qwen3.8-27B吗

尊敬的开发者您好，请使用镜像中心vLLM-MetaX 0.25.0镜像

sglang什么时候支持Qwen3.8-27B-W8A8

尊敬的开发者您好，请使用sglang 0.5.13镜像

APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:38610 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:41270 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:41274 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:41280 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:46244 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:46254 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 509, in adjust_request
(APIServer pid=47) request = self._apply_structural_tag(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/parser/abstract_parser.py", line 535, in _apply_structural_tag
(APIServer pid=47) structure_tag = self._tool_parser.get_structural_tag(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/abstract_tool_parser.py", line 178, in get_structural_tag
(APIServer pid=47) from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/tool_parsers/structural_tag_registry.py", line 12, in <module>
(APIServer pid=47) from xgrammar import StructuralTag, normalize_tool_choice
(APIServer pid=47) ImportError: cannot import name 'normalize_tool_choice' from 'xgrammar' (/opt/conda/lib/python3.12/site-packages/xgrammar/init.py)
(APIServer pid=47) INFO: 127.0.0.1:46256 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=47) ERROR: Exception in ASGI application
(APIServer pid=47) Traceback (most recent call last):
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 422, in run_asgi
(APIServer pid=47) result = await app( # type: ignore[func-returns-value]
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in call
(APIServer pid=47) return await self.app(scope, receive, send)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/applications.py", line 1159, in call
(APIServer pid=47) await super().call(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/applications.py", line 96, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
(APIServer pid=47) await self.app(scope, receive, _send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/cors.py", line 88, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 180, in call
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 178, in call
(APIServer pid=47) await self.app(scope, receive, send_wrapper)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in call
(APIServer pid=47) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 670, in call
(APIServer pid=47) await self.middleware_stack(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 690, in app
(APIServer pid=47) await route.handle(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/routing.py", line 280, in handle
(APIServer pid=47) await self.app(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 134, in app
(APIServer pid=47) await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(APIServer pid=47) raise exc
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
(APIServer pid=47) await app(scope, receive, sender)
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
(APIServer pid=47) response = await f(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 674, in app
(APIServer pid=47) raw_response = await run_endpoint_function(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
(APIServer pid=47) return await dependant.call(values)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 91, in wrapper
(APIServer pid=47) return handler_task.result()
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/serve/utils/api_utils.py", line 112, in wrapper
(APIServer pid=47) return await func(*args, kwargs)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/api_router.py", line 61, in create_chat_completion
(APIServer pid=47) generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 245, in create_chat_completion
(APIServer pid=47) return await self._with_kv_transfer_rejection_cleanup(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/generate/base/serving.py", line 229, in _with_kv_transfer_rejection_cleanup
(APIServer pid=47) return await awaitable
(APIServer pid=47) ^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 266, in _create_chat_completion
(APIServer pid=47) result = await self.render_chat_request(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 231, in render_chat_request
(APIServer pid=47) return await self.online_renderer.render_chat(request)
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 159, in render_chat
(APIServer pid=47) conversation, engine_inputs = await self.preprocess_chat(
(APIServer pid=47) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=47) File "/opt/conda/lib/python3.12/site-packages/vllm/renderers/online_renderer.py", line 413, in preprocess_chat
(APIServer pid=47) ).adjust_request(

容器内报错哦。

尊敬的开发者您好，请pip升级xgrammar

单独升级也是不行的。要改。
xgrannar 0.2.6强制升级 apache-tvn-ffi到0.1.13.post3,破坏了 tilelang的tvm

这是我修改以后的结果，
tvm_ffi 0.1.9(兼容vLLM 0.25.0)
xgrannar 0.2.6 (方 API: nornalize tool choice, get nodel structural tag. openai tool call schema)

升级以后要重新自己打包镜像。

ERROR: Cannot install apache-tvm-ffi==0.1.9 and xgrammar==0.2.6 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested apache-tvm-ffi==0.1.9
xgrammar 0.2.6 depends on apache-tvm-ffi>=0.1.10

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

你好，请问你解决了吗？方便看一下你的pip list结果吗

normalize_tool_choice 是 xgrammar 库的一个公共函数，用于规范化工具调用（tool-calling）的参数。现在的问题是，只要不涉及工具调用，是可以工作的。一旦涉及工具调用，就会报错。但是升级的话，涉及@15287246311同学提到的这个问题，依赖冲突太多了。

强行升级，然后自己重新打包镜像就可以了。

最新的这个developer.metax-tech.com/softnova/docker?chip_name=%E6%9B%A6%E4%BA%91C500%E7%B3%BB%E5%88%97&package_name=vllm-metax:0.27.0-maca.ai3.8.2.302-torch2.10-py312-ubuntu22.04-amd64
就能搞定。速度很顶。