Repository navigation
[Feature] Enable remote inference endpoints for examples #1973
Description
Priority
P1-Stopper
OS type
Ubuntu
Hardware type
Xeon-GNR
Running nodes
Single Node
Description
Story for users
- for Xeon users, users can try bigger models on remote endpoint with Gaudi while bigger models might not work well on Xeon
- for Xeon users, if users want to try on Gaudi to understand the difference between Xeon and Gaudi, they can use remote endpoint
- for Gaudi users, we assume that users have local Gaudi access to run GenAIExamples.
- target Denvr and IBM for remote endpoints in the GenAIExamples
Enable remote inference endpoints for
-
AgentQnA : Sin, Alex
-
Productivity Suite : (Sri?) (P0)
-
ChatQnA : (Sri?) (P0)
-
DocSum : (Sri?) (P0)
-
CodeGen : (Sri?) (P0)
-
FinanceAgent : Alex
-
workflowExecAgent : Louie
-
CodeTrans : Alex
-
AudioQnA : Alex
Models are not supported in current public remote endpoint. ON HOLD for now.
- VideoQnA : (?)
- MulitmodalQnA : Alex
- VisualQnA: Alex
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
louie-tsai commentedon May 27, 2025
@alexsin368 needs to be in assignees list
joshuayao commentedon May 28, 2025
@louie-tsai Ok, I will add @alexsin368 to the repo. Then, we can add him into the assignees list.
@alexsin368 Could you please accept the repo invitation in GitHub?
alexsin368 commentedon May 28, 2025
accepted, thanks
4 remaining items
louie-tsai commentedon Jun 27, 2025
also need to have landing page for Denvr/IBM remote endpoint instructions.
@alexsin368 take AR to ask Megana
louie-tsai commentedon Jun 27, 2025
block at #2005. once remote endpoint online, sample owners need to change test script for CI/CD to use that remote endpoint.
edlee123 commentedon Jul 8, 2025
ChatQnA is done , looking for reviewers please: #2091
alexsin368 commentedon Jul 8, 2025
@joshuayao why did you close this when several PRs linked to this feature are still open?
louie-tsai commentedon Jul 8, 2025
still need to finish the CI/CD for the PRs
edlee123 commentedon Jul 8, 2025
I see, it's related to #2005
For the docker version of ChatQnA endpoint example, do we want something similar to below but connecting to an endpoint instead?
https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/tests/test_compose_on_xeon.sh
When I wrote the test for the llms textgen endpoint, I spun up an OpenAI-compatible vllm endpoint and set an arbitrary key "testkey":
https://github.com/opea-project/GenAIComps/blob/main/tests/llms/test_llms_textgen_endpoint_openai.sh
I also tried with openrouter.ai (it has free models with rate limit), but I didn't use this endpoint in the CI/CD test.
louie-tsai commentedon Jul 17, 2025
@alexsin368 @srinarayan-srikanthan Amber asked us to remove Denvr info here and assume that users will install Enterprise Inference themselves.
louie-tsai commentedon Aug 19, 2025
@joshuayao all PRs are merged for this feature request.
edlee123 commentedon Aug 26, 2025
Hi can someone review please #2091 also to do with ChatQnA using endpoints? Not all openAI Compatible endpoints have the same formatting as Denvr (even if OpenAI compatible) - the #2091 PR allows more endpoints e.g. OpenRouter, HuggingFace, vLLM, and uses the updated text-generation component. This will enable hundreds llm endpoints for ChatQnA.