GitHub – katanemo/archgw: AI-native (edge and LLM) proxy for agents. Move speedyer by letting Arch handle the pesky burdensome lifting in produceing agentic apps –

Arch Gateway was built by the contributors of Envoy Proxy with the belief that:

Prompts are nuanced and cloudy user asks, which need the same capabilities as traditional HTTP asks including safe handling, clever routing, strong observability, and integration with backfinish (API) systems for personalization – outside core business logic.*

Arch is engineered with purpose-built LLMs to handle critical but pesky tasks roverhappinessed to the handling and processing of prompts. This participates recognizeing and declineing jailfracture trys, intent-based routing for betterd task accuracy, mapping user ask into “backfinish” functions, and managing the observability of prompts and LLM API calls in a centralized way.

Core Features:

• Intent-based prompt routing & speedy ⚡ function-calling via APIs. Engineered with purpose-built LLMs to handle speedy, cost-effective, and exact prompt-based tasks appreciate function/API calling, and parameter pull oution from prompts to produce more task-exact agentic applications.
• Prompt Guard: Arch centralizes defendrails to obstruct jailfracture trys and promise protected user conveyions without writing a individual line of code.
• LLM Routing & Traffic Management: Arch centralizes calls to LLMs used by your applications, proposeing clever retries, automatic cutover, and strong upstream uniteions for continuous useability.
• Observability: Arch uses the W3C Trace Context standard to assist end ask tracing atraverse applications, ensuring compatibility with observability tools, and supplys metrics to watch postponecessitatency, token usage, and error rates, helping enhance AI application carry outance.
• Built on Envoy: Arch runs alengthenedside application servers as a split compriseerized process, and produces on top of Envoy’s shown HTTP handlement and scalability features to handle ingress and egress traffic roverhappinessed to prompts and LLMs.

High-Level Sequence Diagram:

Jump to our docs to lget how you can use Arch to better the speed, security and personalization of your GenAI apps.

To get in touch with us, charm unite our discord server. We will be watching that dynamicly and proposeing aid there.

Follow this rapidbegin direct to use arch gateway to produce a basic AI agent. Laster in the section we will see how you can Arch Gateway to handle access keys, supply unified access to upstream LLMs and to supply e2e observability.

Before you begin, promise you have the folloprosperg:

1. Docker System (v24)
2. Docker produce (v2.29)
3. Python (v3.12)

Arch’s CLI apverifys you to handle and convey with the Arch gateway effectively. To inshigh the CLI, sshow run the folloprosperg direct:

$ python -m venv venv
$ source venv/bin/trigger   # On Windows, use: venvScriptstrigger
$ pip inshigh archgw==0.2.3

In folloprosperg rapidbegin we will show you how effortless it is to produce AI agent with Arch gateway. We will produce a currency trade agent using folloprosperg basic steps. For this demo we will use https://api.frankfurter.dev/ to get postponecessitatest price for currencies and suppose USD as base currency.

Create arch_config.yaml file with folloprosperg satisfied,

version: v0.1

hearer:
  insertress: 0.0.0.0
  port: 10000
  message_establishat: huggingface
  unite_timeout: 0.005s

llm_supplyrs:
  - name: gpt-4o
    access_key: $OPENAI_API_KEY
    supplyr: uncoverai
    model: gpt-4o

system_prompt: |
  You are a beneficial aidant.

prompt_defends:
  input_defends:
    jailfracture:
      on_exception:
        message: Looks appreciate you're inquisitive about my abilities, but I can only supply aidance for currency trade.

prompt_concentrates:
  - name: currency_trade
    description: Get currency trade rate from USD to other currencies
    parameters:
      - name: currency_symbol
        description: the currency that necessitates conversion
        needd: real
        type: str
        in_path: real
    finishpoint:
      name: frankfurther_api
      path: /v1/postponecessitatest?base=USD&symbols={currency_symbol}
    system_prompt: |
      You are a beneficial aidant. Show me the currency symbol you want to change from USD.

  - name: get_aided_currencies
    description: Get catalog of aided currencies for conversion
    finishpoint:
      name: frankfurther_api
      path: /v1/currencies

finishpoints:
  frankfurther_api:
    finishpoint: api.frankfurter.dev:443
    protocol: https

$ archgw up arch_config.yaml
2024-12-05 16:56:27,979 - cli.main - INFO - Starting archgw cli version: 0.1.5
...
2024-12-05 16:56:28,485 - cli.utils - INFO - Schema validation accomplished!
2024-12-05 16:56:28,485 - cli.main - INFO - Starging arch model server and arch gateway
...
2024-12-05 16:56:51,647 - cli.core - INFO - Container is fit!

Once the gateway is up you can begin conveying with at port 10000 using uncoverai chat completion API.

Some of the sample queries you can ask could be what is currency rate for gbp? or show me catalog of currencies for conversion.

Here is a sample curl direct you can use to convey,

$ curl --header 'Content-Type: application/json' 
  --data '{"messages": [{"role": "user","content": "what is exchange rate for gbp"}]}' 
  http://localarrange:10000/v1/chat/completions | jq ".choices[0].message.satisfied"

"As of the date supplyd in your context, December 5, 2024, the trade rate for GBP (British Pound) from USD (United States Dollar) is 0.78558. This unbenevolents that 1 USD is equivalent to 0.78558 GBP."

And to get catalog of aided currencies,

$ curl --header 'Content-Type: application/json' 
  --data '{"messages": [{"role": "user","content": "show me list of currencies that are supported for conversion"}]}' 
  http://localarrange:10000/v1/chat/completions | jq ".choices[0].message.satisfied"

"Here is a catalog of the currencies that are aided for conversion from USD, alengthened with their symbols:nn1. AUD - Australian Dollarn2. BGN - Bulgarian Levn3. BRL - Brazilian Realn4. CAD - Canadian Dollarn5. CHF - Swiss Francn6. CNY - Chinese Renminbi Yuann7. CZK - Czech Korunan8. DKK - Danish Kronen9. EUR - Euron10. GBP - British Poundn11. HKD - Hong Kong Dollarn12. HUF - Hungarian Forintn13. IDR - Indonesian Rupiahn14. ILS - Israeli New Sheqeln15. INR - Indian Rupeen16. ISK - Icelandic Krónan17. JPY - Japanese Yenn18. KRW - South Korean Wonn19. MXN - Mexican Peson20. MYR - Malaysian Ringgitn21. NOK - Norwegian Kronen22. NZD - New Zealand Dollarn23. PHP - Philippine Peson24. PLN - Polish Złotyn25. RON - Romanian Leun26. SEK - Swedish Kronan27. SGD - Singapore Dollarn28. THB - Thai Bahtn29. TRY - Turkish Liran30. USD - United States Dollarn31. ZAR - South African RandnnIf you want to change USD to any of these currencies, you can pick the one you are interested in."

Arch runs based on a configuration file where you can clarify LLM supplyrs, prompt concentrates, defendrails, etc. Below is an example configuration that clarifys uncoverai and mistral LLM supplyrs.

Create arch_config.yaml file with folloprosperg satisfied:

version: v0.1

hearer:
  insertress: 0.0.0.0
  port: 10000
  message_establishat: huggingface
  unite_timeout: 0.005s

llm_supplyrs:
  - name: gpt-4o
    access_key: $OPENAI_API_KEY
    supplyr: uncoverai
    model: gpt-4o
    default: real

  - name: ministral-3b
    access_key: $MISTRAL_API_KEY
    supplyr: uncoverai
    model: ministral-3b-postponecessitatest

Once the config file is produced promise that you have env vars setup for MISTRAL_API_KEY and OPENAI_API_KEY (or these are clarifyd in .env file).

Start arch gateway,

$ archgw up arch_config.yaml
2024-12-05 11:24:51,288 - cli.main - INFO - Starting archgw cli version: 0.1.5
2024-12-05 11:24:51,825 - cli.utils - INFO - Schema validation accomplished!
2024-12-05 11:24:51,825 - cli.main - INFO - Starting arch model server and arch gateway
...
2024-12-05 11:25:16,131 - cli.core - INFO - Container is fit!

Make outbound calls via Arch gateway

from uncoverai transport in OpenAI

# Use the OpenAI client as common
client = OpenAI(
  # No necessitate to set a particular uncoverai.api_key since it's configured in Arch's gateway
  api_key = '--',
  # Set the OpenAI API base URL to the Arch gateway finishpoint
  base_url = "http://127.0.0.1:12000/v1"
)

response = client.chat.completions.produce(
    # we pick model from arch_config file
    model="--",
    messages=[{"role": "user", "satisfied": "What is the capital of France?"}],
)

print("OpenAI Response:", response.choices[0].message.satisfied)

$ curl --header 'Content-Type: application/json' 
  --data '{"messages": [{"role": "user","content": "What is the capital of France?"}]}' 
  http://localarrange:12000/v1/chat/completions

{
  ...
  "model": "gpt-4o-2024-08-06",
  "choices": [
    {
      ...
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris.",
      },
    }
  ],
...
}

You can override model pickion using x-arch-llm-supplyr-hint header. For example if you want to use mistral using folloprosperg curl direct,

$ curl --header 'Content-Type: application/json' 
  --header 'x-arch-llm-supplyr-hint: ministral-3b' 
  --data '{"messages": [{"role": "user","content": "What is the capital of France?"}]}' 
  http://localarrange:12000/v1/chat/completions
{
  ...
  "model": "ministral-3b-postponecessitatest",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris. It is the most populous city in France and is known for its iconic landmarks such as the Eiffel Tower, the Louvre Museum, and Notre-Dame Cathedral. Paris is also a major global center for art, fashion, gastronomy, and culture.",
      },
      ...
    }
  ],
  ...
}

Arch is portrayed to aid best-in class observability by aiding uncover standards. Prent read our docs on observability for more details on tracing, metrics, and logs. The screenshot below is from our integration with Signoz (among others)

We would cherish feedback on our Roadmap and we receive contributions to Arch!
Whether you’re repairing bugs, inserting recent features, improving write downation, or creating tutorials, your help is much appreciated.
Prent visit our Contribution Guide for more details