Русский
EnglishCryptocurrencies:
8086 /
Markets: 109515
Market Cap: $ 2 680 993 449 326 /
24h Vol: $ 72 444 019 587 /
BTC
Dominance: 60.043120200776%
Новости
[Перевод] С новым годом: GPT в 500 строках на SQL
В минувшем году все только и говорили об ИИ и о том, как он может всё за вас сделать.
Мне нравится, когда кто-то или что-то работает за меня. Поэтому решил: попрошу-ка ChatGPT написать за меня новогодний пост:
"Эй, ChatGPT. А ты можешь реализовать большую языковую модель на SQL?"
"Нет, SQL не подходит для реализации больших языковых моделей. Язык SQL предназначен для выполнения запросов к данным, хранящимся в РСУБД и для управления этими запросами. Для реализации большой языковой модели требуются сложные алгоритмы, обширные датасеты, а зачастую – и фреймворки для глубокого обучения. SQL всеми этими возможностями не обладает."
Что ж, лишний раз убеждаешься, что, если хочешь что-то сделать хорошо – сделай это сам.
Давайте же воодушевимся этим оптимистическим планом и реализуем большую языковую модель на языке SQL.
Теория
Готовя этот пост, я вдохновлялся чудесной статьёй GPT in 60 Lines of NumPy, которую написал Джей Моди. В этой статье объяснено внутреннее устройство модели GPT, причём, это сделано гораздо лучше, чем могло бы получиться у меня. Поэтому для порядка вкратце напомню контекст.
Что представляет собой генеративная большая языковая модель с технической точки зрения?
Генеративная LLM – это функция. Она принимает на ввод текстовую последовательность символов (в терминологии ИИ она называется «промпт») и возвращает массив строк и чисел. Вот какова сигнатура этой функции:
llm(prompt: str) -> list[tuple[str, float]]Это детерминированная функция. У неё под капотом выполняется серьёзная математика, но вся эта математика жёстко вшита в систему. Если вы будете многократно сообщать этой функции один и тот же ввод, то всякий раз будете получать от неё один и тот же вывод.
Тем, кому доводилось пользоваться ChatGPT и подобными продуктами, это может показаться удивительным, ведь система может давать разные ответы на один и тот же вопрос. Да, всё верно. Чуть ниже объясню, как это устроено.
Что за значения возвращает эта функция?
Нечто подобное:
llm("I wish you a happy New")
0 (' Year', 0.967553)
1 (' Years', 0.018199688)
2 (' year', 0.003573329)
3 (' York', 0.003114716)
4 (' New', 0.0009022804)
…
50252 (' carbohyd', 2.3950911e-15)
50253 (' volunte', 2.2590102e-15)
50254 ('pmwiki', 1.369229e-15)
50255 (' proport', 1.1198108e-15)
50256 (' cumbers', 7.568147e-17)Она возвращает массив кортежей. Каждый кортеж состоит из слова (скорее, строки) и числа. Число — это вероятность, с которой данное слово может оказаться промпте на следующей позиции. Модель «думает», что за фразой «I wish you a happy New…» («Поздравляю тебя с Новым…») с вероятностью 96,7% последует строка «Year» (…годом) – и так далее.
Слово «думает» выше поставлено в кавычках, так как, естественно, модель ни о чём не думает. Она механически возвращает массивы слов и чисел в соответствии с некоторой жёстко прописанной внутренней логикой.
Если эта система такая топорная и детерминированная, как же ей удаётся генерировать разные тексты?
Большие языковые модели используются в текстовых приложениях (чатботы, генераторы контента, кодинг-ассистенты, т.д.). Эти приложения раз за разом вызывают модель и выбирают подсказанное ею слово (с некоторой степенью произвольности). Последнее предложенное слово добавляется к промпту, и модель вызывается снова. Этот цикл продолжается до тех пор, пока не будет сгенерировано достаточное количество слов.
Накопленная таким образом последовательность слов выглядит как текст на естественном языке, с полноценной грамматикой, синтаксисом и даже мнимыми признаками интеллекта и логики. В данном отношении языковые модели подобны марковским цепям, работающим по такому же принципу.
На внутреннем уровне большая языковая модель устроена так, что каждое последующее предлагаемое слово естественным образом продолжало промпт, с учётом имеющейся грамматики, семантики и тональности. Оснастить функцию такой логикой стало возможно благодаря целой серии научных прорывов (и каторжной работе программистов). Удалось разработать целое семейство алгоритмов под названием GPT или «генеративные предобученные трансформеры».
Что такое "генеративный предобученный трансформер"
«Генеративный» означает, что алгоритм генерирует текст (рекурсивно достраивая продолжения к промпту как было показано выше).
«Трансформер» означает, что алгоритм использует конкретный тип нейронной сети. Сеть-трансформер была разработана компанией Google и описана в этой статье.
Термин «предобученный» требует небольшого исторического экскурса. Изначально считалось, что способность модели достраивать текст – это всего лишь предпосылка для получения более глубоких логических выводов в рамках специализированных задач (нужно улавливать логическую связь между фразами), классификации (например, угадать, сколько звёзд у отеля, проанализировав отзывы постояльцев), машинного перевода и т.д. Считалось, что две эти части алгоритма требуется обучать отдельно, и языковая часть позиционировалась как предобучение, подготовка к последующему обучению решать реальную задачу.
Как сформулировано в исходной статье по GPT:
Продемонстрировано, что возможно добиться больших успехов при решении таких задач путём генеративного предобучения языковой модели на разнородном корпусе неразмеченного текста с последующей тонкой настройкой различных параметров для различных слоёв сети (discriminative fine-tuning) в контексте каждой конкретной задачи.
Такие представления сохранялись до тех пор, пока не стало понятно, что для достаточно большой модели второй шаг зачастую излишен. Модель-трансформер, обученная исключительно генерации текстов, оказалась в состоянии без дополнительного обучения следовать человеческим инструкциям, содержащимся в этих текстах. Таким образом, не требовалось никакой дополнительной тонкой настройки.
Итак, с этими вопросами мы разобрались. Теперь давайте сосредоточимся на реализации.
Генерация
Вот что происходит, если мы пытаемся сгенерировать текст на основе промпта при помощи GPT2:
def generate(prompt: str) -> str:
# Преобразует строку в список токенов.
tokens = tokenize(prompt) # tokenize(prompt: str) -> list[int]
while True:
# Выполняет алгоритм.
# Возвращает вероятностные значения для токенов: список из 50257 чисел с плавающей точкой, в сумме дающих 1.
candidates = gpt2(tokens) # gpt2(tokens: list[int]) -> list[float]
# Выбирает следующий токен из списка предложенных
next_token = select_next_token(candidates)
# select_next_token(candidates: list[float]) -> int
# Прикрепляет его к списку токенов
tokens.append(next_token)
# Решаем, хотим ли мы прекратить генерацию.
# Здесь может быть счётчик токенов, задержка, стоп-слово или что-то ещё.
if should_stop_generating():
break
# Преобразует список токенов в строку
completion = detokenize(tokens) # detokenize(tokens: list[int]) -> str
return completionДавайте реализуем все эти элементы один за другим на языке SQL.
Токенизатор
Прежде, чем текст можно будет скормить нейронке, его требуется преобразовать в список чисел. Конечно, нас это не удивляет: именно такие задачи решаются при помощи текстовых кодировок вроде Unicode. Но обычный Unicode не слишком хорошо работает с нейронными сетями.
В сущности, нейронные сети построены на обширном перемножении матриц, и вся прогностическая сила нейронных сетей заключена в коэффициентах этих матриц. Некоторые из таких матриц содержат по строке на каждое из возможных значений в «алфавите», в других предусмотрено по строке на «символ».
В этом контексте смысл терминов «алфавит» и «символ» требуется пояснить. В Unicode длина «алфавита» составляет 149186 символов (именно столько отдельных точек Unicode существует на момент подготовки этой статьи), а в качестве «символа» может выступать, например: ﷽ (да, это одиночная точка Unicode, номер 65021, она соответствует целой фразе на арабском языке, особенно важной для мусульман). Отмечу, что эту фразу можно было бы записать и обычной арабской вязью. Таким образом, одному тексту может соответствовать несколько вариантов кодировки.
Давайте разберём эту ситуацию на примере слова "PostgreSQL". Если бы мы попытались закодировать её (преобразовать в массив чисел) при помощи Unicode, то у нас получилось бы 10 чисел, которые могли бы находиться в диапазоне от 1 до 149186. Таким образом, в нашей нейронной сети требовалось бы хранить матрицу из 149186 строк и совершить ряд вычислений над 10 строками из этой матрицы. Некоторые из этих строк (соответствующие буквам латиницы) использовались бы особенно часто, и в них было бы заключено много информации. Другие символы, например, смайлик какашки и таинственные символы из мёртвых языков едва ли вообще хоть раз использовались, но и на их хранение требуется место.
Естественно, мы хотим, чтобы оба эти числа, и длина «алфавита», и количество «символов» оставались как можно меньше. В идеале «символы» нашего алфавита должны быть распределены равномерно, а ещё мы по-прежнему хотим, чтобы наша кодировка не уступала по мощности Unicode.
Интуитивно понятно, что это можно сделать, присвоив уникальные номера тем последовательностям слов, которые часто встречаются в обрабатываемых нами текстах. В Unicode вышеупомянутая религиозная арабская фраза может быть закодирована либо как один кодовый символ, либо буква за буквой. Поскольку мы выкатываем нашу собственную кодировку, точно так же мы можем поступать с теми словами и фразами, которые важны в рамках нашей модели (например, они часто встречаются в текстах).
Например, можно было бы выделить конкретные числа для строк "Post", "greSQL" и "ing". Таким образом, оба слова "PostgreSQL" и "Posting" в нашем представлении будут иметь длину по 2. Конечно же, мы стремимся закреплять отдельные кодовые символы для более конкретных последовательностей и отдельных байт. Даже натыкаясь на тарабарщину или на иноязычный текст, такую информацию всё равно удастся закодировать, хотя, на это и потребуется больше времени.
В GPT2 используется вариант алгоритма под названием кодирование диадами (Byte Pair Encoding), предназначенный специально для этой цели. Токенизатор GPT2 использует словарь из 50257 кодовых точек (в терминологии ИИ — «токенов»), которые соответствуют различным последовательностям байт в UTF-8 (плюс отдельный токен, означающий «конец текста»).
Этот словарь мы собрали путём статистического анализа, выполненного таким образом:
1. Начинаем с простой кодировки из 256 токенов: по токену за байт.
2. Берём большой корпус текстов (предпочтительно именно тот, на котором будет обучаться модель).
3. Кодируем его.
4. Вычисляем, какая пара токенов встречается чаще всего. Пусть это будет 0x20 0x74 (пробел, за которым следует строчная "t").
5. Присваиваем следующее доступное значение (257) этой паре байт.
6. Повторяем шаги 3-5, теперь обращая внимание на последовательности байт. Если последовательность байт можно закодировать сложным токеном, то используем сложный токен. При явных неоднозначностях (например, "abc" может в одном случае быть закодировано как "a" + "bc" или как "ab" + "c"), то пользуйтесь той, номер которой меньше (это означает, что она была добавлена раньше и, следовательно, встречается чаще). Это нужно делать рекурсивно, пока все последовательности, которые можно сложить в один токен, будут сложены в один токен.
7. Выполним такое «складывание» 50000 раз подряд.
Число 50000 выбрано разработчиками более или менее произвольно. В других моделях количество токенов также держится в подобном диапазоне (от 30k до 100k).
На каждой итерации данного алгоритма в словарь будет добавляться новый токен, получаемый конкатенацией двух предыдущих. В конечном итоге получим 50256 токенов. Добавляем токен с фиксированным номером, означающий «конец текста» - и всё готово.
В версии алгоритма BTE для GPT2 есть ещё один уровень кодировки: в словаре токенов эти токены отображаются на строки, а не на массивы байт. Отображение с байт на строковые символы определяется в этой функции. Тот словарь, который она производит, находится в таблице encoder.
Рассмотрим, как реализовать токенизатор на SQL.
Данный токенизатор – неотъемлемая часть GPT2, и словарь токенов можно скачать с сайта OpenAI, равно как и оставшуюся часть модели. Нам понадобится импортировать её в таблицу tokenizer. В самом конце этого поста вы найдёте ссылку на репозиторий с кодом. Этот код автоматизирует заполнение таблиц базы данных, необходимых для работы с моделью.
В рамках рекурсивного подхода CTE разбиваем слово на токены (начиная с единичных байт) и объединяем наилучшие смежные пары, пока больше материала для объединения не останется. Само объединение происходит по принципу вложенного рекурсивного CTE.
Для примера я выбрал слово «Mississippilessly». Каждая запись в наборе результатов соответствует наилучшей найденной к данному моменту паре, которую можно «сложить», а также прогресс в обработке запроса.
WITH RECURSIVE
bpe AS
(
SELECT (n + 1)::BIGINT AS position, character, TRUE AS continue, 1 AS step,
NULL::INT AS token, NULL::TEXT AS combined
FROM CONVERT_TO('Mississippilessly', 'UTF-8') AS bytes
CROSS JOIN LATERAL
GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
JOIN encoder
ON byte = GET_BYTE(bytes, n)
UNION ALL
(
WITH RECURSIVE
base AS
(
SELECT *
FROM bpe
WHERE continue
),
bn AS
(
SELECT ROW_NUMBER() OVER (ORDER BY position) AS position,
continue,
character,
character || LEAD(character) OVER (ORDER BY position) AS cluster
FROM base
),
top_rank AS
(
SELECT tokenizer.*
FROM bn
CROSS JOIN LATERAL
(
SELECT *
FROM tokenizer
WHERE tokenizer.cluster = bn.cluster
LIMIT 1
) tokenizer
ORDER BY
token
LIMIT 1
),
breaks AS
(
SELECT 0::BIGINT AS position, 1 AS length
UNION ALL
SELECT bn.position,
CASE WHEN token IS NULL THEN 1 ELSE 2 END
FROM breaks
JOIN bn
ON bn.position = breaks.position + length
LEFT JOIN
top_rank
USING (cluster)
)
SELECT position, character, token IS NOT NULL,
(SELECT step + 1 FROM base LIMIT 1), token, top_rank.cluster
FROM breaks
LEFT JOIN
top_rank
ON 1 = 1
CROSS JOIN LATERAL
(
SELECT STRING_AGG(character, '' ORDER BY position) AS character
FROM bn
WHERE bn.position >= breaks.position
AND bn.position < breaks.position + length
) bn
WHERE position > 0
)
)
SELECT step, MAX(token) AS token, MAX(combined) AS combined, ARRAY_AGG(character ORDER BY position)
FROM bpe
WHERE continue
GROUP BY
step
ORDER BY
step
На каждом шаге алгоритм BPE находит пару токенов, наиболее подходящую для слияния, и объединяет их (здесь в выводе вы видите такую слитую пару и ее ранг). При помощи такой процедуры размер пространства токенов можно сократить с 150k, присущих Unicode, до 50k, а количество токенов в данном конкретном слове с 17 до 5. В обоих случаях это отличное улучшение.
Когда приходится работать с множеством слов, токенизатор сначала разбивает текст на отдельные слова при помощи этого регулярного выражения и объединяет токены в каждом слове отдельно. К сожалению, при работе с PostgreSQL свойства символов Unicode не поддерживаются на уровне регулярных выражений, поэтому мне пришлось немного отладить систему (возможно, в процессе я угробил нормальную поддержку Unicode). Вот как результат выглядит в SQL:
WITH input AS
(
SELECT 'PostgreSQL is great' AS prompt
),
clusters AS
(
SELECT part_position, bpe.*
FROM input
CROSS JOIN LATERAL
REGEXP_MATCHES(prompt, '''s|''t|''re|''ve|''m|''ll|''d| ?\w+| ?\d+| ?[^\s\w\d]+|\s+(?!\S)|\s+', 'g') WITH ORDINALITY AS rm (part, part_position)
CROSS JOIN LATERAL
(
WITH RECURSIVE
bpe AS
(
SELECT (n + 1)::BIGINT AS position, character, TRUE AS continue
FROM CONVERT_TO(part[1], 'UTF-8') AS bytes
CROSS JOIN LATERAL
GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
JOIN encoder
ON byte = GET_BYTE(bytes, n)
UNION ALL
(
WITH RECURSIVE
base AS
(
SELECT *
FROM bpe
WHERE continue
),
bn AS
(
SELECT ROW_NUMBER() OVER (ORDER BY position) AS position,
continue,
character,
character || LEAD(character) OVER (ORDER BY position) AS cluster
FROM base
),
top_rank AS
(
SELECT tokenizer.*
FROM bn
CROSS JOIN LATERAL
(
SELECT *
FROM tokenizer
WHERE tokenizer.cluster = bn.cluster
LIMIT 1
) tokenizer
ORDER BY
token
LIMIT 1
),
breaks AS
(
SELECT 0::BIGINT AS position, 1 AS length
UNION ALL
SELECT bn.position,
CASE WHEN token IS NULL THEN 1 ELSE 2 END
FROM breaks
JOIN bn
ON bn.position = breaks.position + length
LEFT JOIN
top_rank
USING (cluster)
)
SELECT position, character, token IS NOT NULL
FROM breaks
LEFT JOIN
top_rank
ON 1 = 1
CROSS JOIN LATERAL
(
SELECT STRING_AGG(character, '' ORDER BY position) AS character
FROM bn
WHERE bn.position >= breaks.position
AND bn.position < breaks.position + length
) bn
WHERE position > 0
)
)
SELECT position, character AS cluster
FROM bpe
WHERE NOT continue
) bpe
),
tokens AS
(
SELECT token, cluster
FROM clusters
JOIN tokenizer
USING (cluster)
)
SELECT *
FROM tokens
Странный символ Ġ соответствует пробелу.
Этот запрос токенизирует промпт и преобразует его в массив чисел. Вот промпт и готов к долгому пути сквозь слои модели.
Векторные представления
Токены - это представления единиц естественного языка (в целом, примерно 0,75 слова на токен), поэтому любая модель, претендующая на успешное автозавершение текстов, должна каким-то образом кодировать отношения между этими единицами. Даже в изоляции части речи обладают наборами ортогональных свойств.
Возьмём, к примеру, слово «subpoena» (повестка) – оказывается, в токенизаторе GPT2 на него отведён целый токен. С точки зрения английского языка – это существительное? Да, бесспорно. А глагол? Да, и глаголом может быть. А прилагательным? Уже не очень, но, в принципе, контекст придумать можно. Это канцелярит? Чёрт возьми, ещё какой. И так далее.
Все эти свойства ортогональны, то есть, не зависят друг от друга. Слово может быть официозным существительным, но не глаголом и не прилагательным. В английском возможны любые сочетания этих признаков.
Сущности с ортогональными свойствами лучше всего кодируются в виде векторных представлений. Пусть у нас будет не единственное свойство (например, номер), а много. При желании мы можем крутить ими как захотим. Например, фраза «В судебном решении, которое процитировал адвокат, фигурирует…» с высокой вероятностью продолжается каким-то существительным, которое, не менее вероятно, стилистически относится к канцеляриту. Нас совершенно не интересует, есть ли у этого слова маргинальная семантика «глагол», либо «прилагательное», либо «цветок».
В математике отображение сравнительно узких значений на более широкие пространства (например, ID токенов на векторы) называется векторным представлением. Именно этим мы здесь и занимаемся.
Как решить, какие именно свойства окажутся представлены в этих векторах? А мы не будем. Мы просто предоставим достаточно векторного пространства для каждого токена и понадеемся, что модель на этапе обучения получит достаточно информации, чтобы заполнить все эти измерения осмысленным содержимым. В GPT2 под векторы отводится 768 измерений. Заранее (и, на самом деле, даже в ретроспективе) не известно, какое свойство слова будет закодировано, например, в измерении 247. Определённо, что-то закодировано там будет, но что именно – сказать сложно.
Какие свойства каждого из токенов мы внедрим в векторное пространство? Любые, которые помогут нам спрогнозировать, каков будет следующий токен.
ID токена? Естественно. Разные токены означают разные вещи.
Позиция токена в тексте? Да, пожалуйста. «Сине-фиолетовый» и «фиолетово-синий» — не одно и то же.
Взаимные отношения токенов? Естественно! Это, пожалуй, наиважнейшая часть задачи, и блок внимания (Attention) в архитектуре трансформера как раз впервые позволил решить эту часть задачи правильно.
Токены и позиции легко поддаются векторному представлению. Допустим, есть фраза «PostgreSQL is great», о которой заранее известно, что она отображается на четыре токена: [6307, 47701, 318, 1049].
Среди прочих параметров GPT2 есть две матрицы, называющиеся WTE (векторное представление лексических токенов) и WPE (векторное представление позиций слов). Как понятно из названия, в первом хранятся векторные представления токенов, а во втором – векторные представления их позиций. Конкретные значения, которыми окажутся заполнены эти представления, набираются в результате обучения модели, в данном случае GPT2. Насколько известно, в таблицах баз данных wte и wpe есть константы. Размер
WTE равен 50257×768, а размер WPE равен 1024×768. Второе означает, что в GPT2 можно использовать промпт, состоящий не более чем из 1024 токенов. Если записать в промпт больше токенов, мы просто не сможем вытянуть для токенов сверх 1024 их позиционные представления. Это архитектурный аспект (в терминологии ИИ – «гиперпараметр») модели, устанавливаемый на этапе её проектирования, и обучением его изменить невозможно. Когда говорят о «контекстном окне» большой языковой модели, имеют в виду именно это число.
Токен 6307 стоит у нас на месте 0, токен 47701 на месте 1, токен 318 на месте 2, а токен 1049 на месте 3. Для каждого из этих токенов и их позиций у нас по два вектора: один из WTE и один из WPE. Их требуется сложить. В результате получатся четыре вектора, которые послужат вводом для следующей части алгоритма: так работает нейронная сеть прямого распространения, в которой действует механизм внимания.
Для работы с SQL мы воспользуемся pgvector, это расширение PostgreSQL.
Небольшая оговорка: код для этого поста я пишу на обычном SQL, иногда вставляю чистые SQL-функции в качестве вспомогательных элементов. В рамках этого поста осуществить такое не составит труда, равно как определить векторные операции над массивами. Это делается за счёт небольшого снижения производительности (и уже было реализовано в версии 1 и работало, пусть и медленно). С распространением ИИ и ростом влияния векторных баз данных pgvector или эквивалентный механизм определённо укоренится в ядре PostgreSQL в течение двух-трёх релизов. Я просто решил оседлать волну, которая принесёт меня в будущее.
Вот как это делается в SQL:
WITH embeddings AS
(
SELECT place, values
FROM UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
)
SELECT place, (values::REAL[])[0:5]
FROM embeddings
(Чтобы вывод был короче, в этом запросе показаны только первые 5 измерений для каждого вектора).
Внимание
Та часть трансформерной архитектуры, благодаря которой она действительно феерит, называется «механизм внутреннего внимания» (self-attention). Он был впервые описан в 2017 году в статье «Attention is all you need» под авторством Васмани и др. Пожалуй, это наиизвестнейшая статья об искусственном интеллекте, и с тех пор её название превратилось в болванку (клише для именования других статей).
Итак, вероятно, к настоящему моменту у нас уже есть несколько векторов, в которых закодированы определённые синтаксические и семантические свойства слов, содержащихся в нашем промпте. Нам каким-то образом нужно донести эти свойства до последнего вектора. Сейчас будет небольшой спойлер: именно в последнем векторе будет храниться представление того слова, которым продолжится фраза.
Во фразе вида a «I looked at the violet and saw that it was not the usual …» (я взглянул на лиловое пятно и заметил, что это не обычный…), на месте многоточия должно быть что-то, увиденное вами (об этом свидетельствует слово «saw»), обладающее свойством «лиловости» и при этом «необычное» (сочетаем токены «не» и «обычный», тем самым меняя знак измерения «обычность» на противоположный). Можно привести такую реалистичную аналогию: человек читает книгу на иностранном языке, о котором имеет некоторое представление, но свободно им не владеет. В таком случае человеку придётся аккуратно сплетать каждое слово со следующим, и, если он не обратит внимания на ключевую часть фразы, то всю фразу поймёт неправильно.
Чтобы обеспечить такой перенос значения от одного токена к другому, требуется разрешить векторам всех токенов влиять друг на друга. Так, возвращаясь к английскому: если мы хотим наполнить слово «it» конкретной семантикой, то какая часть семантики должна поступить от предыдущих векторов в промпте, а какая – остаться от самого слова «it»?
В данной модели эта задача решается при помощи 12 наборов матриц, именуемых Q (запрос), K (ключ) и V (значение). В каждой из них по 64 столбца. Их мы получаем из векторных представлений через 768×2304-мерное линейное преобразование c_attn, веса и смещения которого хранятся в таблицах c_attn_w и c_attn_b.
Результатом c_attn является матрица со n_token строк и 2304 столбцами (3×12×64). Она состоит из 12 Q-матриц, 12 K-матриц и 12 V-матриц, упорядоченных горизонтально именно в таком порядке.
Каждый набор, состоящий из Q, K и V, называется «головой». С их помощью выполняется шаг под названием «многоголовое условное внутреннее внимание», на котором вычисляется функция внимания.
Она вычисляется по этой формуле:
где softmax – это функция нормализации весов. Она определяется так:
M - это матрица-константа, именуемая «условной маской». Она определяется так:
Функция Softmax обнуляет все отрицательные бесконечности.
Почему необходимо использовать маски?
В промпте из предыдущего примера было 4 токена, и модель первым делом вычисляла 4 векторных представления для 4 этих токенов. По мере дальнейшей работы модели над этими векторами будет выполнено много вычислений, но в основном они пойдут независимо и параллельно. Изменения в одном векторе не затронут другие векторы, как если бы иных векторов не существовало. Векторы влияют друг на друга только в пределах блока внутреннего внимания.
Как только модель справится с математикой, претенденты на роль следующего токена будут выбираться исключительно исходя из последнего векторного представления. Вся информация должна направляться именно к этому последнему вектору, а не прочь от него. Временные значения последнего вектора не должны влиять на временные значения предыдущих векторов, использовавшихся в ходе прямого распространения по данной модели.
Именно поэтому мы «маскируем» позднейшие векторы, чтобы они не влияли на более ранние через этот конкретный канал. В этом и заключается условность «многоголового условного внутреннего внимания».
Почему матрицы называются "запрос", "ключ" и "значение"?
Честно говоря, даже не уверен, что это хорошая аналогия. Но, тем не менее, попытаюсь объяснить, какова логика таких названий.
Как правило, в машинном обучении при вычислениях не должны использоваться циклы переменной длины или ветвление инструкций. Все задачи должны решаться компоновкой простых аналитических функций (сложение, умножение, возведение в степень, логарифмы и тригонометрия). Так обеспечивается обратное распространение для эффективной реализации которого требуются такие технологии, как автоматическое дифференцирование.
Математическая модель хранилища ключей и значений заключена в таком выражении:
Но это не гладкая дифференцируемая функция, и с обратным распространением она работать не будет. Чтобы формула работала, необходимо получить гладкую функцию, которая была бы близка к v, когда k близка к q, и близка к 0 в иных случаях.
Для этой цели отлично подходит Гауссово распределение («колоколообразная кривая») "bell curve"), отмасштабированная к v с ожиданием k и достаточно небольшим стандартным отклонением:
где σ - произвольный параметр, определяющий, насколько «остра» колоколообразная кривая.
Если взять в векторном пространстве с достаточно большим количеством измерений фиксированный вектор K и несколько векторов Q, которые случайным и равномерным образом отклоняются от K в каждом из измерений, то их скалярные произведения сложатся именно в колоколообразную кривую. Соответственно, в векторном пространстве «дифференцируемое хранилище ключей и значений» можно смоделировать при помощи выражения
того самого, которым мы пользовались в нашей функции внимания.
Опять же, эта аналогия немного натянута. Лучше не слишком зацикливаться на концептуальной составляющей внимания, то есть, на потоке операций, хеш-таблицах и пр. Можете считать, что они просто вдохновили меня на описанный математический фокус, который я догадался протестировать – и на практике он оказался весьма хорош.
Проиллюстрирую этот шаг:
WITH embeddings AS
(
SELECT place, values
FROM UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = 0
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = 0
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = 0
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = 0
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM embeddings
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
head AS
(
SELECT place,
(values::REAL[])[1:64]::VECTOR(64) AS q,
(values::REAL[])[1 + 768:64 + 768]::VECTOR(64) AS k,
(values::REAL[])[1 + 1536:64 + 1536]::VECTOR(64) AS v
FROM mha_norm
),
sm_input AS
(
SELECT h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM head h1
CROSS JOIN
head h2
),
sm_diff AS
(
SELECT x, y, value - MAX(value) OVER (PARTITION BY x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT x, y AS place, e / SUM(e) OVER (PARTITION BY x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, (ARRAY_AGG(value ORDER BY ordinality))[:3] AS values
FROM (
SELECT x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * head.v) AS values
FROM softmax
JOIN head
USING (place)
GROUP BY
x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
)
SELECT place,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((q::REAL[])[:3]) AS n) AS q,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((k::REAL[])[:3]) AS n) AS k,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((v::REAL[])[:3]) AS n) AS v,
matrix,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:3]) AS n) AS attention
FROM head
JOIN attention
USING (place)
JOIN (
SELECT x AS place, STRING_AGG(CASE WHEN value > 0 THEN TO_CHAR(value, '0.00') ELSE ' 0' END, ' ' ORDER BY place) AS matrix
FROM softmax
GROUP BY
x
) softmax_grouped
USING (place)
Вот что мы сделали:
1. Прежде, чем вычислять функцию внимания, мы нормализовали векторы при помощи линейного преобразования R’ = RГ1 + B1. Матрица Г1 и вектор B1называются, соответственно, «масштабирование» и «сдвиг». Это параметры, изученные моделью, которые хранятся в таблицах ln_1_g и ln_1_b
2. Мы показываем только первую голову первого слоя, используемого в этом алгоритме. После того, как умножим векторы на изученные коэффициенты из c_attn_w и c_attn_b («вес» и «смещение»), порежем полученные в результате 2304-мерные векторы. Возьмём 64-мерные векторы, начинающиеся в позициях 0, 768 и 1536. Они соответствуют векторам Q, K и V первой головы.
3. EXP в PostgreSQL плохо работает с очень малыми числами, и именно поэтому мы прибегаем к нулю, если EXP получает аргумент менее -745,13.
4. Для каждого вектора мы показываем только первые три элемента. Марицу внимания показываем полностью.
Итак, значение первого вектора копируется в вывод как есть (так мы поступаем с первым значением на каждом слое алгоритма). Это означает, что, когда модель уже обучена, результирующее векторное представление первого токена будет определяться только значением первого токена. В принципе, на этапе рекурсивного вывода, пока токены только добавляются в промпт, по сравнению с предыдущей итерацией изменится лишь самое последнее векторное представление в выводе. Именно поэтому применяется условная маска.
Заглянем немного вперёд: блок внимания – это единственное место во всём алгоритме, где токены могут влиять друг на друга на этапе прямого прохода. Поскольку на данном этапе мы не позволяем позднейшим токенам влиять на более раннее (и отключили такую возможность), все вычисления, проделанные над более ранними токенами, можно повторно использовать на разных итерациях прямого прохода по модели.
Напомню, модель при работе добавляет токены в промпт. Если исходный (токенизированный) промпт – это «Post greSQL Ġis Ġgreat», а следующий, например, будет «Post greSQL Ġis Ġgreat Ġfor», то все результаты вычислений, проделанных над первыми четырьмя токенами, можно будет повторно использовать и в новом промпте. Они никогда не изменятся, независимо от того, что ещё будет прикреплено к этому промпту.
В статье Джея Муди, которую я привёл для примера, этот аспект не задействуется (равно как и в моей статье – для простоты), но в оригинальной реализации GPT2 он задействован.
Закончив работу со всеми головами, получаем 12 матриц. Каждая из них будет по 64 столбца в ширину и по n_tokens строк в высоту. Чтобы вновь отобразить её на размерность векторных представлений (768), нам просто нужно составить эти матрицы горизонтально.
Последний этап обработки многоголового внимания связан с проецированием значений путём линейного преобразования в том же самом измерении. Соответствующие веса и смещения хранятся в таблицах c_proj_w и c_proj_b.
Ниже приведён полный код, описывающий реализацию многоголового внимания в первом слое:
WITH embeddings AS
(
SELECT place, values
FROM UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
c_proj_w AS
(
SELECT *
FROM c_proj_w
WHERE block = 0
),
c_proj_b AS
(
SELECT *
FROM c_proj_b
WHERE block = 0
),
mlp_c_fc_w AS
(
SELECT *
FROM mlp_c_fc_w
WHERE block = 0
),
mlp_c_fc_b AS
(
SELECT *
FROM mlp_c_fc_b
WHERE block = 0
),
mlp_c_proj_w AS
(
SELECT *
FROM mlp_c_proj_w
WHERE block = 0
),
mlp_c_proj_b AS
(
SELECT *
FROM mlp_c_proj_b
WHERE block = 0
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = 0
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = 0
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = 0
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = 0
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM embeddings
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
heads AS
(
SELECT place, head,
(values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
(values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
(values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
FROM mha_norm
CROSS JOIN
GENERATE_SERIES(0, 11) head
),
sm_input AS
(
SELECT head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM heads h1
JOIN heads h2
USING (head)
),
sm_diff AS
(
SELECT head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
FROM (
SELECT head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
FROM softmax
JOIN heads
USING (head, place)
GROUP BY
head, x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
),
mha AS
(
SELECT place, w.values + c_proj_b.values AS values
FROM (
SELECT attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
FROM attention
CROSS JOIN
c_proj_w
GROUP BY
attention.place
) w
CROSS JOIN
c_proj_b
)
SELECT place,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM mha
Прежде чем результаты многоголового внимания будут переданы на следующий шаг, к ним нужно добавить оригинальные входные данные. Этот фокус был описан ещё в исходной статье по трансформерам. Предполагается, что он должен помочь с затухающими и взрывными градиентами.
Это распространённая проблема при обучении: иногда градиенты параметров оказываются слишком малы или слишком велики. Если менять их при очередной итерации обучения, это также очень слабо влияет на функцию потерь (поэтому модель сходится очень медленно). Бывает и обратный эффект: даже минимальное изменение отбрасывает функцию потерь слишком далеко от локального минимума, тем самым обесценивая всю работу по обучению.
Прямое распространение
Именно в этом и заключается работа глубоких нейронных сетей. Большая часть параметров модели на самом деле используется именно на этом этапе.
Здесь мы имеем дело с многослойным перцептроном, у которого три слоя (768, 3072, 768), а в качестве функции активации используем линейную единицу гауссовской ошибки (GELU):
Как показывает практика, эта функция даёт очень хорошие результаты при работе с глубокими нейронными сетями. Её можно аналитически аппроксимировать вот так:
Изученные параметры линейного преобразования для связей в слое называются c_fc (768 → 3072) и c_proj (3072 → 768). Значения с первого слоя сначала нормализуются при помощи коэффициентов, содержащихся в изученном параметре ln_2. По завершении этапа прямого распространения его ввод вновь добавляется к выводу. Этот элемент также входит в самый первый вариант дизайна трансформеров. Весь этап прямого распространения выглядит так:
А вот как это делается на SQL:
WITH embeddings AS
(
SELECT place, values
FROM UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
c_proj_w AS
(
SELECT *
FROM c_proj_w
WHERE block = 0
),
c_proj_b AS
(
SELECT *
FROM c_proj_b
WHERE block = 0
),
mlp_c_fc_w AS
(
SELECT *
FROM mlp_c_fc_w
WHERE block = 0
),
mlp_c_fc_b AS
(
SELECT *
FROM mlp_c_fc_b
WHERE block = 0
),
mlp_c_proj_w AS
(
SELECT *
FROM mlp_c_proj_w
WHERE block = 0
),
mlp_c_proj_b AS
(
SELECT *
FROM mlp_c_proj_b
WHERE block = 0
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = 0
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = 0
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = 0
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = 0
),
ln_2_b AS
(
SELECT *
FROM ln_2_b
WHERE block = 0
),
ln_2_g AS
(
SELECT *
FROM ln_2_g
WHERE block = 0
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM embeddings
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
heads AS
(
SELECT place, head,
(values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
(values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
(values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
FROM mha_norm
CROSS JOIN
GENERATE_SERIES(0, 11) head
),
sm_input AS
(
SELECT head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM heads h1
JOIN heads h2
USING (head)
),
sm_diff AS
(
SELECT head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
FROM (
SELECT head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
FROM softmax
JOIN heads
USING (head, place)
GROUP BY
head, x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
),
mha AS
(
SELECT place, w.values + c_proj_b.values + embeddings.values AS values
FROM (
SELECT attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
FROM attention
CROSS JOIN
c_proj_w
GROUP BY
attention.place
) w
CROSS JOIN
c_proj_b
JOIN embeddings
USING (place)
),
ffn_norm AS
(
SELECT place, agg.values * ln_2_g.values + ln_2_b.values AS values
FROM (
SELECT place, norm.values
FROM mha
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_2_b
CROSS JOIN
ln_2_g
),
ffn_a AS
(
SELECT gelu.place, gelu.values
FROM (
SELECT place, w.values + mlp_c_fc_b.values AS values
FROM (
SELECT ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
FROM ffn_norm
CROSS JOIN
mlp_c_fc_w
GROUP BY
ffn_norm.place
) w
CROSS JOIN
mlp_c_fc_b
) v
CROSS JOIN LATERAL
(
SELECT place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
GROUP BY
place
) gelu
),
ffn AS
(
SELECT place, w.values + mlp_c_proj_b.values + mha.values AS values
FROM (
SELECT ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
FROM ffn_a
CROSS JOIN
mlp_c_proj_w
GROUP BY
ffn_a.place
) w
CROSS JOIN
mlp_c_proj_b
JOIN mha
USING (place)
)
SELECT place,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM ffn
Именно этот вывод получаем из первого блока GPT2.
Блоки
Операции, рассмотренные на предыдущих этапах, повторяются в каждом из слоёв (именуемых «блоками»). Блоки выстраиваются в виде конвейера, так, что вывод из предыдущего блока сразу подаётся на ввод следующему. У каждого из блоков – свой набор изученных параметров.
В SQL потребуется соединить блоки при помощи рекурсивного CTE.
Как только финальный блок выдаст значения, результат потребуется нормализовать при помощи изученного параметра ln_f.
Вот как в итоге будет выглядеть модель:
А вот как результат выглядит на SQL:
WITH RECURSIVE
initial AS
(
SELECT ARRAY[6307, 47701, 318, 1049] AS input
),
hparams AS
(
SELECT 12 AS n_block
),
embeddings AS
(
SELECT place, values
FROM initial
CROSS JOIN
hparams
CROSS JOIN LATERAL
UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
transform AS
(
SELECT 0 AS block, place, values
FROM embeddings
UNION ALL
(
WITH previous AS
(
SELECT *
FROM transform
)
SELECT block + 1 AS block, transformed_layer.*
FROM hparams
CROSS JOIN LATERAL
(
SELECT block
FROM previous
WHERE block < 12
LIMIT 1
) q
CROSS JOIN LATERAL
(
WITH ln_2_b AS
(
SELECT *
FROM ln_2_b
WHERE block = q.block
),
ln_2_g AS
(
SELECT *
FROM ln_2_g
WHERE block = q.block
),
c_proj_w AS
(
SELECT *
FROM c_proj_w
WHERE block = q.block
),
c_proj_b AS
(
SELECT *
FROM c_proj_b
WHERE block = q.block
),
mlp_c_fc_w AS
(
SELECT *
FROM mlp_c_fc_w
WHERE block = q.block
),
mlp_c_fc_b AS
(
SELECT *
FROM mlp_c_fc_b
WHERE block = q.block
),
mlp_c_proj_w AS
(
SELECT *
FROM mlp_c_proj_w
WHERE block = q.block
),
mlp_c_proj_b AS
(
SELECT *
FROM mlp_c_proj_b
WHERE block = q.block
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = q.block
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = q.block
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = q.block
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = q.block
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM previous
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
heads AS
(
SELECT place, head,
(values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
(values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
(values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
FROM mha_norm
CROSS JOIN
GENERATE_SERIES(0, 11) head
),
sm_input AS
(
SELECT head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM heads h1
JOIN heads h2
USING (head)
),
sm_diff AS
(
SELECT head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
FROM (
SELECT head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
FROM softmax
JOIN heads
USING (head, place)
GROUP BY
head, x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
),
mha AS
(
SELECT place, w.values + c_proj_b.values + previous.values AS values
FROM (
SELECT attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
FROM attention
CROSS JOIN
c_proj_w
GROUP BY
attention.place
) w
CROSS JOIN
c_proj_b
JOIN previous
USING (place)
),
ffn_norm AS
(
SELECT place, agg.values * ln_2_g.values + ln_2_b.values AS values
FROM (
SELECT place, norm.values
FROM mha
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_2_b
CROSS JOIN
ln_2_g
),
ffn_a AS
(
SELECT gelu.place, gelu.values
FROM (
SELECT place, w.values + mlp_c_fc_b.values AS values
FROM (
SELECT ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
FROM ffn_norm
CROSS JOIN
mlp_c_fc_w
GROUP BY
ffn_norm.place
) w
CROSS JOIN
mlp_c_fc_b
) v
CROSS JOIN LATERAL
(
SELECT place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
GROUP BY
place
) gelu
),
ffn AS
(
SELECT place, w.values + mlp_c_proj_b.values + mha.values AS values
FROM (
SELECT ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
FROM ffn_a
CROSS JOIN
mlp_c_proj_w
GROUP BY
ffn_a.place
) w
CROSS JOIN
mlp_c_proj_b
JOIN mha
USING (place)
)
SELECT *
FROM ffn
) transformed_layer
)
),
block_output AS
(
SELECT *
FROM hparams
JOIN transform
ON transform.block = n_block
),
ln_f AS
(
SELECT place, norm.values * ln_f_g.values + ln_f_b.values AS values
FROM block_output
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) AS n(value)
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
) norm
CROSS JOIN
ln_f_b
CROSS JOIN
ln_f_g
)
SELECT place,
(SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM ln_f
Это вывод модели.
Четвёртый вектор – это и есть фактическое представление следующего токена, предсказанного моделью. Нам просто нужно отобразить его обратно на токены.
Токены
У нас уже есть векторное представление (768-мерное), в котором, согласно модели, заключены семантика и грамматика наиболее вероятного продолжения промпта. Теперь нам нужно отобразить его обратно на токен.
Одна из первых операций, совершаемых моделью – это отображение токенов на их векторные представления. Эта операция делается при помощи матрицы wpe размером 50257×768. Той же самой матрицей нам потребуется воспользоваться, чтобы отобразить векторное представление обратно на токен.
Проблема в том, что строго обратное отображение выполнить невозможно: (вероятно) векторное представление не будет совпадать ни с одной из строк в матрице. Поэтому нам придётся найти токен, который «ближе всего» к векторному представлению.
Поскольку (как мы надеемся), в измерениях векторных представлений заключены некоторые семантические и грамматические аспекты токена, необходимо сопоставить их как можно точнее. Один из способов консолидировать «степень близости» каждого из измерений – вычислить скалярное произведение двух векторных представлений. Чем выше будет это значение, тем ближе токен к прогнозу.
Для этого умножим векторное представление на матрицу wte. В результате получим матрицу из одного столбца и 50257 строк. Каждое значение в этом результате будет произведением спрогнозированного вектора и векторного представления токена. Чем выше это число, тем вероятнее, что токен станет продолжением промпта.
Чтобы выбрать следующий токен, нужно просто преобразовать значения сходства в значения вероятности. Для этого нам пригодится старая добрая функция softmax (та самая, при помощи которой мы уже нормализовали веса внимания).
Почему стоит использовать softmax для работы с вероятностями?
Softmax – это удобное свойство аксиомы выбора Льюса. Оно таково: относительная вероятность двух вариантов не зависит от наличия прочих вероятностных вариантов. Если вариант A влвое вероятнее варианта B, то это соотношение не зависит от наличия или отсутствия других вариантов (хотя, конечно, абсолютные значения при этом могут измениться).
Вектор скалярных произведений («логит» в терминологии ИИ) содержит произвольные значения, которые могут не укладываться в какую-либо шкалу. Если балл A выше, чем балл B, то известно, что A вероятнее – а больше ничего не известно. Можно подкорректировать входные значения softmax так, как нам угодно, главное, соблюдать их порядок (более крупные значения остаются более крупными).
Распространённый подход, позволяющий этого добиться – нормализация значений путём вычитания наибольшего значения из каждого значения в множестве (так что наибольшее значение становится равным 0, а все остальные – отрицательным числам). Затем берём фиксированное количество наивысших значений (скажем, топ-5 или топ-10). Наконец, перед передачей всех значений softmax умножаем их на константу.
Количество наивысших значений, с которым мы работаем, обычно называется top_n, а множитель-константа (вернее, обратное ему число) называется «температурой» (Т). Чем выше температура, тем более гладкое распределение вероятностей у нас получится, и тем выше вероятность, что на место продолжения будет выбран не первый попавшийся токен.
Формула для расчёта вероятностей токенов
где scores — это множество из наивысших top_n баллов.
Почему этот показатель называется "температурой"
Функция softmax также называется распределение Больцмана. Она широко используется в физике. Среди прочего, на ней основана барометрическая формула, по которой рассчитывается изменение плотности воздуха в зависимости от высоты.
Как известно, тёплый воздух поднимается вверх и распределяется на сравнительно большой высоте. В горячем воздухе сравнительно высока вероятность, что молекула ль частых столкновений с другими молекулами заберётся на такую высоту, куда в ином случае бы не попала. В отличие от ситуации с более холодными температурами, плотность воздуха увеличивается на больших высотах и падает на уровне моря.
Сравните, как воздух движется при разных температурах:
\Иллюстрация Доминика Форда, Bouncing Balls and the Boltzmann Distribution
Аналогично, в рассматриваемом нами примере, чем больше «температура», тем выше вероятность, что будет выбран не самый очевидный токен (разумеется, за счёт наиболее очевидных). Логический вывод становится менее предсказуемым и более «творческим».
Давайте выразим всё это на SQL. У нас был промпт «PostgreSQL is great». Вот первые 5 токенов, которые, в соответствии с этой моделью, являются наиболее вероятными продолжениями этой фразы. Показаны их вероятности при разной температуре:
WITH RECURSIVE
initial AS
(
SELECT ARRAY[6307, 47701, 318, 1049] AS input
),
hparams AS
(
SELECT 12 AS n_block,
5 AS top_n,
ARRAY_LENGTH(input, 1) AS n_seq
FROM initial
),
embeddings AS
(
SELECT place, values
FROM initial
CROSS JOIN
hparams
CROSS JOIN LATERAL
UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
transform AS
(
SELECT 0 AS block, place, values
FROM embeddings
UNION ALL
(
WITH previous AS
(
SELECT *
FROM transform
)
SELECT block + 1 AS block, transformed_layer.*
FROM hparams
CROSS JOIN LATERAL
(
SELECT block
FROM previous
WHERE block < 12
LIMIT 1
) q
CROSS JOIN LATERAL
(
WITH ln_2_b AS
(
SELECT *
FROM ln_2_b
WHERE block = q.block
),
ln_2_g AS
(
SELECT *
FROM ln_2_g
WHERE block = q.block
),
c_proj_w AS
(
SELECT *
FROM c_proj_w
WHERE block = q.block
),
c_proj_b AS
(
SELECT *
FROM c_proj_b
WHERE block = q.block
),
mlp_c_fc_w AS
(
SELECT *
FROM mlp_c_fc_w
WHERE block = q.block
),
mlp_c_fc_b AS
(
SELECT *
FROM mlp_c_fc_b
WHERE block = q.block
),
mlp_c_proj_w AS
(
SELECT *
FROM mlp_c_proj_w
WHERE block = q.block
),
mlp_c_proj_b AS
(
SELECT *
FROM mlp_c_proj_b
WHERE block = q.block
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = q.block
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = q.block
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = q.block
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = q.block
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM previous
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
heads AS
(
SELECT place, head,
(values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
(values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
(values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
FROM mha_norm
CROSS JOIN
GENERATE_SERIES(0, 11) head
),
sm_input AS
(
SELECT head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM heads h1
JOIN heads h2
USING (head)
),
sm_diff AS
(
SELECT head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
FROM (
SELECT head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
FROM softmax
JOIN heads
USING (head, place)
GROUP BY
head, x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
),
mha AS
(
SELECT place, w.values + c_proj_b.values + previous.values AS values
FROM (
SELECT attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
FROM attention
CROSS JOIN
c_proj_w
GROUP BY
attention.place
) w
CROSS JOIN
c_proj_b
JOIN previous
USING (place)
),
ffn_norm AS
(
SELECT place, agg.values * ln_2_g.values + ln_2_b.values AS values
FROM (
SELECT place, norm.values
FROM mha
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_2_b
CROSS JOIN
ln_2_g
),
ffn_a AS
(
SELECT gelu.place, gelu.values
FROM (
SELECT place, w.values + mlp_c_fc_b.values AS values
FROM (
SELECT ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
FROM ffn_norm
CROSS JOIN
mlp_c_fc_w
GROUP BY
ffn_norm.place
) w
CROSS JOIN
mlp_c_fc_b
) v
CROSS JOIN LATERAL
(
SELECT place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
GROUP BY
place
) gelu
),
ffn AS
(
SELECT place, w.values + mlp_c_proj_b.values + mha.values AS values
FROM (
SELECT ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
FROM ffn_a
CROSS JOIN
mlp_c_proj_w
GROUP BY
ffn_a.place
) w
CROSS JOIN
mlp_c_proj_b
JOIN mha
USING (place)
)
SELECT *
FROM ffn
) transformed_layer
)
),
block_output AS
(
SELECT *
FROM hparams
JOIN transform
ON transform.block = n_block
),
ln_f AS
(
SELECT place, norm.values * ln_f_g.values + ln_f_b.values AS values
FROM block_output
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) AS n(value)
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
) norm
CROSS JOIN
ln_f_b
CROSS JOIN
ln_f_g
),
logits AS
(
SELECT logits.*
FROM hparams
CROSS JOIN LATERAL
(
SELECT token, INNER_PRODUCT(ln_f.values, wte.values) AS value
FROM ln_f
CROSS JOIN
wte
WHERE ln_f.place = n_seq - 1
ORDER BY
value DESC
LIMIT (top_n)
) logits
),
temperatures (temperature) AS
(
VALUES
(0.5),
(1),
(2)
),
tokens AS
(
SELECT token, value, softmax, temperature
FROM temperatures
CROSS JOIN LATERAL
(
SELECT *, (e / SUM(e) OVER ()) AS softmax
FROM (
SELECT *,
(value - MAX(value) OVER ()) / temperature AS diff
FROM logits
) exp_x
CROSS JOIN LATERAL
(
SELECT CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
) exp
) q
)
SELECT token,
cluster,
TO_CHAR(t1.value, 'S00.000') AS score,
TO_CHAR(t1.softmax, '0.00') AS "temperature = 0.5",
TO_CHAR(t2.softmax, '0.00') AS "temperature = 1",
TO_CHAR(t3.softmax, '0.00') AS "temperature = 2"
FROM (
SELECT *
FROM tokens
WHERE temperature = 0.5
) t1
JOIN (
SELECT *
FROM tokens
WHERE temperature = 1
) t2
USING (token)
JOIN (
SELECT *
FROM tokens
WHERE temperature = 2
) t3
USING (token)
JOIN tokenizer
USING (token)
Логический вывод
Вот мы и готовы перейти к реальному логическому выводу: запустить модель, выбрать токен в соответствии с полученной вероятностью, добавить токен к промпту и повторять этот процесс до тех пор, пока не будет сгенерировано достаточное количество токенов.
Как было показано выше, сама большая языковая модель является детерминированной. Это просто последовательные перемножения матриц и другие математические операции, производимые над заранее определёнными константами. Пока промпт и такие гиперпараметры как температура и топ-N остаются неизменными, вывод также остаётся неизменным.
Единственный недетерминированный процесс в данном случае – это выбор токена. В нём заложена (переменная) степень случайности. Вот почему чатботы, основанные на GPT, могут давать разные ответы на один и тот же промпт.
Воспользуемся в качестве промпта фразой «Happy New Year! I wish» и прикажем модели сгенерировать 10 новых токенов для этого промпта. Температуру установим в 2, а top_n в 5.
У меня на компьютере на этот запрос уходит 2 минуты и 44 секунды. Вот такой вывод получается:
SELECT SETSEED(0.20231231);
WITH RECURSIVE
input AS
(
SELECT 'Happy New Year! I wish you' AS prompt,
10 AS threshold,
2 AS temperature,
1 AS top_n
),
clusters AS
(
SELECT part_position, bpe.*
FROM input
CROSS JOIN LATERAL
REGEXP_MATCHES(prompt, '''s|''t|''re|''ve|''m|''ll|''d| ?\w+| ?\d+| ?[^\s\w\d]+|\s+(?!\S)|\s+', 'g') WITH ORDINALITY AS rm (part, part_position)
CROSS JOIN LATERAL
(
WITH RECURSIVE
bpe AS
(
SELECT (n + 1)::BIGINT AS position, character, TRUE AS continue
FROM CONVERT_TO(part[1], 'UTF-8') AS bytes
CROSS JOIN LATERAL
GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
JOIN encoder
ON byte = GET_BYTE(bytes, n)
UNION ALL
(
WITH RECURSIVE
base AS
(
SELECT *
FROM bpe
WHERE continue
),
bn AS
(
SELECT ROW_NUMBER() OVER (ORDER BY position) AS position,
continue,
character,
character || LEAD(character) OVER (ORDER BY position) AS cluster
FROM base
),
top_rank AS
(
SELECT tokenizer.*
FROM bn
CROSS JOIN LATERAL
(
SELECT *
FROM tokenizer
WHERE tokenizer.cluster = bn.cluster
LIMIT 1
) tokenizer
ORDER BY
token
LIMIT 1
),
breaks AS
(
SELECT 0::BIGINT AS position, 1 AS length
UNION ALL
SELECT bn.position,
CASE WHEN token IS NULL THEN 1 ELSE 2 END
FROM breaks
JOIN bn
ON bn.position = breaks.position + length
LEFT JOIN
top_rank
USING (cluster)
)
SELECT position, character, token IS NOT NULL
FROM breaks
LEFT JOIN
top_rank
ON 1 = 1
CROSS JOIN LATERAL
(
SELECT STRING_AGG(character, '' ORDER BY position) AS character
FROM bn
WHERE bn.position >= breaks.position
AND bn.position < breaks.position + length
) bn
WHERE position > 0
)
)
SELECT position, character AS cluster
FROM bpe
WHERE NOT continue
) bpe
),
tokens AS
(
SELECT ARRAY_AGG(token ORDER BY part_position, position) AS input
FROM clusters
JOIN tokenizer
USING (cluster)
),
gpt AS
(
SELECT input, ARRAY_LENGTH(input, 1) AS original_length
FROM tokens
UNION ALL
SELECT input || next_token.token, original_length
FROM gpt
CROSS JOIN
input
CROSS JOIN LATERAL
(
WITH RECURSIVE
hparams AS
(
SELECT ARRAY_LENGTH(input, 1) AS n_seq,
12 AS n_block
),
embeddings AS
(
SELECT place, values
FROM hparams
CROSS JOIN LATERAL
UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
CROSS JOIN LATERAL
(
SELECT ordinality - 1 AS place
) o
CROSS JOIN LATERAL
(
SELECT wte.values + wpe.values AS values
FROM wte
CROSS JOIN
wpe
WHERE wte.token = tokens.token
AND wpe.place = o.place
) embedding
),
transform AS
(
SELECT 0 AS block, place, values
FROM embeddings
UNION ALL
(
WITH previous AS
(
SELECT *
FROM transform
)
SELECT block + 1 AS block, transformed_layer.*
FROM hparams
CROSS JOIN LATERAL
(
SELECT block
FROM previous
WHERE block < 12
LIMIT 1
) q
CROSS JOIN LATERAL
(
WITH ln_2_b AS
(
SELECT *
FROM ln_2_b
WHERE block = q.block
),
ln_2_g AS
(
SELECT *
FROM ln_2_g
WHERE block = q.block
),
c_proj_w AS
(
SELECT *
FROM c_proj_w
WHERE block = q.block
),
c_proj_b AS
(
SELECT *
FROM c_proj_b
WHERE block = q.block
),
mlp_c_fc_w AS
(
SELECT *
FROM mlp_c_fc_w
WHERE block = q.block
),
mlp_c_fc_b AS
(
SELECT *
FROM mlp_c_fc_b
WHERE block = q.block
),
mlp_c_proj_w AS
(
SELECT *
FROM mlp_c_proj_w
WHERE block = q.block
),
mlp_c_proj_b AS
(
SELECT *
FROM mlp_c_proj_b
WHERE block = q.block
),
c_attn_w AS
(
SELECT *
FROM c_attn_w
WHERE block = q.block
),
c_attn_b AS
(
SELECT *
FROM c_attn_b
WHERE block = q.block
),
ln_1_g AS
(
SELECT *
FROM ln_1_g
WHERE block = q.block
),
ln_1_b AS
(
SELECT *
FROM ln_1_b
WHERE block = q.block
),
mha_norm AS
(
SELECT place, mm.values + c_attn_b.values AS values
FROM (
SELECT place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
FROM (
SELECT place, agg.values * ln_1_g.values + ln_1_b.values AS values
FROM (
SELECT place, norm.values
FROM previous
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_1_b
CROSS JOIN
ln_1_g
) layer_norm
CROSS JOIN
c_attn_w
GROUP BY
place
) mm
CROSS JOIN
c_attn_b
),
heads AS
(
SELECT place, head,
(values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
(values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
(values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
FROM mha_norm
CROSS JOIN
GENERATE_SERIES(0, 11) head
),
sm_input AS
(
SELECT head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
FROM heads h1
JOIN heads h2
USING (head)
),
sm_diff AS
(
SELECT head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
FROM sm_input
),
sm_exp AS
(
SELECT head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
FROM sm_diff
),
softmax AS
(
SELECT head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
FROM sm_exp
),
attention AS
(
SELECT place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
FROM (
SELECT head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
FROM softmax
JOIN heads
USING (head, place)
GROUP BY
head, x
) q
CROSS JOIN LATERAL
UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
GROUP BY
place
),
mha AS
(
SELECT place, w.values + c_proj_b.values + previous.values AS values
FROM (
SELECT attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
FROM attention
CROSS JOIN
c_proj_w
GROUP BY
attention.place
) w
CROSS JOIN
c_proj_b
JOIN previous
USING (place)
),
ffn_norm AS
(
SELECT place, agg.values * ln_2_g.values + ln_2_b.values AS values
FROM (
SELECT place, norm.values
FROM mha
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) value
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
) norm
) agg
CROSS JOIN
ln_2_b
CROSS JOIN
ln_2_g
),
ffn_a AS
(
SELECT gelu.place, gelu.values
FROM (
SELECT place, w.values + mlp_c_fc_b.values AS values
FROM (
SELECT ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
FROM ffn_norm
CROSS JOIN
mlp_c_fc_w
GROUP BY
ffn_norm.place
) w
CROSS JOIN
mlp_c_fc_b
) v
CROSS JOIN LATERAL
(
SELECT place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
GROUP BY
place
) gelu
),
ffn AS
(
SELECT place, w.values + mlp_c_proj_b.values + mha.values AS values
FROM (
SELECT ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
FROM ffn_a
CROSS JOIN
mlp_c_proj_w
GROUP BY
ffn_a.place
) w
CROSS JOIN
mlp_c_proj_b
JOIN mha
USING (place)
)
SELECT *
FROM ffn
) transformed_layer
)
),
block_output AS
(
SELECT *
FROM hparams
JOIN transform
ON transform.block = n_block
),
ln_f AS
(
SELECT place, norm.values * ln_f_g.values + ln_f_b.values AS values
FROM block_output
CROSS JOIN LATERAL
(
SELECT AVG(value) AS mean,
VAR_POP(value) AS variance
FROM UNNEST(values::REAL[]) AS n(value)
) agg
CROSS JOIN LATERAL
(
SELECT ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
FROM UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
) norm
CROSS JOIN
ln_f_b
CROSS JOIN
ln_f_g
),
logits AS
(
SELECT token, INNER_PRODUCT(ln_f.values, wte.values) AS value
FROM hparams
JOIN ln_f
ON ln_f.place = n_seq - 1
CROSS JOIN
wte
ORDER BY
value DESC
LIMIT (top_n)
),
tokens AS
(
SELECT token,
high - softmax AS low,
high
FROM (
SELECT *,
SUM(softmax) OVER (ORDER BY softmax) AS high
FROM (
SELECT *, (e / SUM(e) OVER ()) AS softmax
FROM (
SELECT *,
(value - MAX(value) OVER ()) / temperature AS diff
FROM logits
) exp_x
CROSS JOIN LATERAL
(
SELECT CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
) exp
) q
) q
),
next_token AS
(
SELECT *
FROM (
SELECT RANDOM() AS rnd
) r
CROSS JOIN LATERAL
(
SELECT *
FROM tokens
WHERE rnd >= low
AND rnd < high
) nt
)
SELECT *
FROM next_token
) next_token
WHERE ARRAY_LENGTH(input, 1) < original_length + threshold
AND next_token.token <> 50256
),
output AS
(
SELECT CONVERT_FROM(STRING_AGG(SET_BYTE('\x00', 0, byte), '' ORDER BY position), 'UTF8') AS response
FROM (
SELECT STRING_AGG(cluster, '' ORDER BY ordinality) AS response
FROM input
JOIN gpt
ON ARRAY_LENGTH(input, 1) = original_length + threshold
CROSS JOIN LATERAL
UNNEST(input) WITH ORDINALITY n (token, ordinality)
JOIN tokenizer
USING (token)
) q
CROSS JOIN LATERAL
STRING_TO_TABLE(response, NULL) WITH ORDINALITY n (character, position)
JOIN encoder
USING (character)
)
SELECT *
FROM output
ИИ со своей работой справился отлично. Я искренне желаю вам всего самого лучшего в наступившем году!
Все запросы и код для установки выложены в следующем репозитории на GitHub: quassnoi/explain-extended-2024.
-
30.06.26 15:08
wendytaylor015
My name is Wendy Taylor, I'm from Los Angeles, i want to announce to you Viewer how Capital Crypto Recover help me to restore my Lost Bitcoin, I invested with a Crypto broker without proper research to know what I was hoarding my hard-earned money into scammers, i lost access to my crypto wallet or had your funds stolen? Don’t worry Capital Crypto Recover is here to help you recover your cryptocurrency with cutting-edge technical expertise, With years of experience in the crypto world, Capital Crypto Recover employs the best latest tools and ethical hacking techniques to help you recover lost assets, unlock hacked accounts, Whether it’s a forgotten password, Capital Crypto Recover has the expertise to help you get your crypto back. a security company service that has a 100% success rate in the recovery of crypto assets, i lost wallet and hacked accounts. I provided them the information they requested and they began their investigation. To my surprise, Capital Crypto Recover was able to trace and recover my crypto assets successfully within 24hours. Thank you for your service in helping me recover my $647,734 worth of crypto funds and I highly recommend their recovery services, they are reliable and a trusted company to any individuals looking to recover lost money. Contact email [email protected] OR Telegram @Capitalcryptorecover Call/Text Number +1 (336)390-6684 his contact: [email protected] His website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
02.07.26 01:22
Lieneke Bonnema
I highly recommend ResQprofirm for their professional asset recovery services of my $120,000 scammed funds. Their expertise, professionalism, and commitment to achieving results make them a reliable choice for anyone seeking dependable recovery assistance. [email protected], WhatsApp +19852969146, telegram @resqprofirm
-
02.07.26 01:26
Clara Morin
I want to extend my deepest appreciation for showing that circumstances do not define one’s potential for greatness. Your support has been a major source of inspiration during my trading journey, and I am sincerely grateful for your insight and mentorship. Thank you so much. [email protected], WhatsApp +19852969146, telegram ResQprofirm
-
02.07.26 01:31
Robin Hale
I sincerely want to thank you for demonstrating that anyone can rise above their circumstances and achieve success. Your constant support has been incredibly inspiring during my trading journey, and your wisdom and advice mean so much to me. I appreciate you deeply. [email protected], WhatsApp +19852969146, telegram Resqprofirm
-
04.07.26 15:32
Fraddy Pual
There are few companies I trust as much as FUNDSRETRIEVER. When I lost $653,000 in Ethereum to a ruthless scam, I thought my life would never be the same. The betrayal cut deep, but I refused to give up. I searched tirelessly for a legitimate way to recover what was stolen, and finally found FUNDSRETRIEVER—the most competent and compassionate recovery team I could have imagined. They handled my case with precision and care, and in the end, my entire ETH wallet was restored. More than the money, they gave me back my hope and happiness. I'm sharing my story because I want others to know that recovery is possible. If a scam has taken from you, don't hesitate—contact FUNDSRETRIEVER today. Email: FUNDSRETRIEVER1@ Gmail.com | WhatsApp: +1 603-512-1448 | Telegram: @FUNDSRETRIEVER
-
05.07.26 14:44
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/Text: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
05.07.26 14:44
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/Text: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
06.07.26 16:20
Olga Ognjanović
Having trouble withdrawing funds from an investment platform? ResQprofirm provides fund recovery assistance for individuals seeking help with investment-related disputes. I reached out to them after experiencing problems with an investment platform, and I appreciated their professionalism and support throughout the process. If you're facing a similar situation, act promptly, keep records of your transactions and communications, and seek assistance from a qualified recovery service or the appropriate authorities. Contact: Email: [email protected] Telegram: @ResQprofirm WhatsApp: +1 985 296 9146
-
06.07.26 16:31
Joseph Weigl
Invest wisely and stay cautious. Don't be influenced by promises of unusually high returns or convincing sales pitches from brokers. I learned this the hard way after falling victim to an investment scam that promised huge profits. Fortunately, I acted quickly and reported the incident to a recovery firm for assistance. Contact: Email: [email protected] Telegram: @Resqprofirm WhatsApp: +1 985 296 9146
-
06.07.26 16:33
Jaran Løvlien
A heartfelt thank you to RESQPRO FIRM for their commitment and professionalism throughout the investigation of my case. Their team worked diligently and helped recover assets valued at $88,000, which were returned to my wallet. I truly appreciate their support, clear communication, and dedication, and I'm grateful for the assistance I received. Contact: Email: [email protected] Telegram: @Resqprofirm WhatsApp: +1 985 296 9146
-
07.07.26 18:00
robertalfred175
CRYPTO SCAM RECOVERY SUCCESSFUL – A TESTIMONIAL OF LOST PASSWORD TO YOUR DIGITAL WALLET BACK. My name is Robert Alfred, Am from Australia. I’m sharing my experience in the hope that it helps others who have been victims of crypto scams. A few months ago, I fell victim to a fraudulent crypto investment scheme linked to a broker company. I had invested heavily during a time when Bitcoin prices were rising, thinking it was a good opportunity. Unfortunately, I was scammed out of $120,000 AUD and the broker denied me access to my digital wallet and assets. It was a devastating experience that caused many sleepless nights. Crypto scams are increasingly common and often involve fake trading platforms, phishing attacks, and misleading investment opportunities. In my desperation, a friend from the crypto community recommended Capital Crypto Recovery Service, known for helping victims recover lost or stolen funds. After doing some research and reading multiple positive reviews, I reached out to Capital Crypto Recovery. I provided all the necessary information—wallet addresses, transaction history, and communication logs. Their expert team responded immediately and began investigating. Using advanced blockchain tracking techniques, they were able to trace the stolen Dogecoin, identify the scammer’s wallet, and coordinate with relevant authorities to freeze the funds before they could be moved. Incredibly, within 24 hours, Capital Crypto Recovery successfully recovered the majority of my stolen crypto assets. I was beyond relieved and truly grateful. Their professionalism, transparency, and constant communication throughout the process gave me hope during a very difficult time. If you’ve been a victim of a crypto scam, I highly recommend them with full confidence contacting: Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text: +1 (336) 390-6684 Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
07.07.26 18:01
robertalfred175
CRYPTO SCAM RECOVERY SUCCESSFUL – A TESTIMONIAL OF LOST PASSWORD TO YOUR DIGITAL WALLET BACK. My name is Robert Alfred, Am from Australia. I’m sharing my experience in the hope that it helps others who have been victims of crypto scams. A few months ago, I fell victim to a fraudulent crypto investment scheme linked to a broker company. I had invested heavily during a time when Bitcoin prices were rising, thinking it was a good opportunity. Unfortunately, I was scammed out of $120,000 AUD and the broker denied me access to my digital wallet and assets. It was a devastating experience that caused many sleepless nights. Crypto scams are increasingly common and often involve fake trading platforms, phishing attacks, and misleading investment opportunities. In my desperation, a friend from the crypto community recommended Capital Crypto Recovery Service, known for helping victims recover lost or stolen funds. After doing some research and reading multiple positive reviews, I reached out to Capital Crypto Recovery. I provided all the necessary information—wallet addresses, transaction history, and communication logs. Their expert team responded immediately and began investigating. Using advanced blockchain tracking techniques, they were able to trace the stolen Dogecoin, identify the scammer’s wallet, and coordinate with relevant authorities to freeze the funds before they could be moved. Incredibly, within 24 hours, Capital Crypto Recovery successfully recovered the majority of my stolen crypto assets. I was beyond relieved and truly grateful. Their professionalism, transparency, and constant communication throughout the process gave me hope during a very difficult time. If you’ve been a victim of a crypto scam, I highly recommend them with full confidence contacting: Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text: +1 (336) 390-6684 Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
09.07.26 19:06
Toivo Walli
I lost 8.56btc to a fake Bitcoin mining site, I tried withdrawing but couldn't approved my process, I reported to !R£SQPROFIRM! via °R£SQproFirm°àt°gmail•com° °tEL£°gram=R£SQprofirm °whaT°Zap+198°52°96°91°46
-
09.07.26 19:10
Misty Alexander
Ongoing messages demanding more money before approving withdrawals are a major red flag. Stop engaging and report the incident to a trusted recovery team. For professional support, you can contact R£sQprofirm using °ResQproFirm°àt°g,*ma'il(•)¢m°, TEL£gram ResQprofirm, or |whaTZap| +1-985-296-9146.
-
09.07.26 19:13
Clara Soto
Anyone receiving continued requests for additional deposits from a scam platform should immediately cut off communication and submit the case to a reputable recovery service for investigation. R£sQprofirm is a dependable firm you can reach at °ResQproFirm°àt°g,*ma'il(•)¢om°, TEL£gram ResQprofirm, or |whaTZap| +1-985-296-9146.
-
12.07.26 03:30
Kora Baltacha
Time is critical. Act now by reaching out to a reputable, seasoned recovery specialist who will guide you every step of the way. You'll need to submit transaction proof, scammer details, and any other useful information. Armed with this, the experts can trace and attempt to pull your money back from the scammers' hidden accounts or wallets. Best of all, R£sQprofirm provides recovery help without charging any upfront fees. Contact them immediately via Telegram @ResQprofirm, WhatsApp +19852969146, or email [email protected].
-
12.07.26 03:33
Pahal Mathew
It's important to move proactively by engaging an experienced recovery specialist. They will assist you throughout the process. To help them, provide: · Transaction evidence · Scammer information · Any additional relevant details The experts will then track and try to retrieve your funds from the scammers' hidden accounts or wallets. R£sQprofirm offers recovery assistance with no upfront fees. Contact: Telegram: @ResQprofirm WhatsApp: +19852969146 Email: [email protected]
-
13.07.26 23:49
[email protected]
One of the biggest concerns I have about cryptocurrency is the lack of regulation. It creates opportunities for scammers to invent convincing stories and fraudulent investment schemes. Unfortunately, some social media platforms continue to display these ads because they profit from them, even after users report them.I personally clicked on a Facebook advertisement for a company called Chickenfastmining and ended up losing more than $120,000 in a scam. I reported the ad, but nothing was done. Later, through a Reddit community, I found a recovery service called CYBERBERSPY that, in my personal experience, they helped me recover $110,000 of my lost funds. If you've been a victim of a cryptocurrency scam, don't lose hope. Explore your options carefully, and always verify the legitimacy of any recovery service before trusting them or paying any fees. Based on my own experience, CYBERBERSPY was helpful to me and i was able to recover my funds back, but I encourage everyone to do their own research before using any recovery service.i highly recommend: ([email protected])
-
15.07.26 11:53
Sarah Green
Thank you for showing that success is possible regardless of where someone starts. Your encouragement, valuable advice, and continuous support have inspired me throughout my $160,457k crypto investment recovery journey. I truly appreciate your kindness and dedication. Resqprofirm @gmail.com Telegram: Resqprofirm
-
15.07.26 11:58
Lily Gagné
I sincerely appreciate you for proving that anyone can overcome challenges and achieve success. Your unwavering support throughout my trading investment scam of $88,890 recovery journey has been truly inspiring, and your guidance and wisdom have meant a great deal to me. Thank you for everything ResQprofirm@ gmail.com, ResQprofirm on the telegram.
-
16.07.26 21:38
patricialovick86
How To Recover Your Bitcoin Without Falling Victim To Scams: A Testimony Experience With Capital Crypto Recover Services, Contact Telegram: @Capitalcryptorecover Dear Everyone, I would like to take a moment to share my positive experience with Capital Crypto Recover Services. Initially, I was unsure if it would be possible to recover my stolen bitcoins. However, with their expertise and professionalism, I was able to fully recover my funds. Unfortunately, many individuals fall victim to scams in the cryptocurrency space, especially those involving fraudulent investment platforms. However, I advise caution, as not all recovery services are legitimate. I personally lost $273,000 worth of Bitcoin from my Binance account due to a deceptive platform. If you have suffered a similar loss, you may be considering crypto recovery, The Capital Crypto Recover is the most knowledgeable and effective Capital Crypto Recovery Services assisted me in recovering my stolen funds within 24 hours, after getting access to my wallet. Their service was not only prompt but also highly professional and effective, and many recovery services may not be trustworthy. Therefore, I highly recommend Capital Crypto Recover to you. i do always research and see reviews about their service, For assistance finding your misplaced cryptocurrency, get in touch with them, They do their jobs quickly and excellently, Stay safe and vigilant in the crypto world. Contact: [email protected] You can reach them via email at [email protected] OR Call/Text Number +1 (336)390-6684 his contact website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
16.07.26 21:38
patricialovick86
How To Recover Your Bitcoin Without Falling Victim To Scams: A Testimony Experience With Capital Crypto Recover Services, Contact Telegram: @Capitalcryptorecover Dear Everyone, I would like to take a moment to share my positive experience with Capital Crypto Recover Services. Initially, I was unsure if it would be possible to recover my stolen bitcoins. However, with their expertise and professionalism, I was able to fully recover my funds. Unfortunately, many individuals fall victim to scams in the cryptocurrency space, especially those involving fraudulent investment platforms. However, I advise caution, as not all recovery services are legitimate. I personally lost $273,000 worth of Bitcoin from my Binance account due to a deceptive platform. If you have suffered a similar loss, you may be considering crypto recovery, The Capital Crypto Recover is the most knowledgeable and effective Capital Crypto Recovery Services assisted me in recovering my stolen funds within 24 hours, after getting access to my wallet. Their service was not only prompt but also highly professional and effective, and many recovery services may not be trustworthy. Therefore, I highly recommend Capital Crypto Recover to you. i do always research and see reviews about their service, For assistance finding your misplaced cryptocurrency, get in touch with them, They do their jobs quickly and excellently, Stay safe and vigilant in the crypto world. Contact: [email protected] You can reach them via email at [email protected] OR Call/Text Number +1 (336)390-6684 his contact website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
17.07.26 19:24
laimqq90
I recommend Marie when it comes to recovering lost/stolen ust/bitcoin or any kind of cryptocurrencies' from fake investment platforms because they're well specialized in that area and you'll get your money back in full. I can boldly say this right now based on my prior deal i had with them; she was the only one who was able to recover my lost money $52,760 dollars back to my account, Only * ([email protected] and WhatsApp +1 7127594675 successful in recovering my money. They are the only one who can fully restore your lost funds to your account without any deductions, I really value their work and am recommending her to you today. THANK ME LATER
-
17.07.26 20:12
martinsjude080
Needs Online Fraud Help Contact Mighty Hacker Recovery https://mightyhackarrecovery.com I lost $292,900 in Bitcoin after investing with an online mining company. After realizing I had been scammed, I spent a long time searching for ways to recover my funds and contacted several services without success. During my research, I came across Mighty Hacker Recovery through Google and YouTube. What caught my attention was that they said they would not require any upfront payment before providing their recovery service. I decided to contact them to discuss my case and understand their process. Throughout the process, they kept me informed about the progress. After several days, I was asked to provide my Bitcoin wallet address, and my case was concluded. Their fee was handled after the service rather than being requested in advance. If you've been the victim of a cryptocurrency scam, it's important to do your own research, ask questions, and carefully evaluate any recovery service before proceeding. Every case is different, so take the time to verify information and understand the process before making any decisions. For Bitcoin scam recovery, cryptocurrency scam, crypto recovery service, recover stolen Bitcoin, Bitcoin fraud help, blockchain investigation, crypto wallet recovery, online investment scam, digital asset recovery, and crypto scam support. Contact Them on WhatsApp +1 (343) 947-3496 or [email protected] or [email protected] or https://mightyhackarrecovery.com Scam recovery Online fraud help Scam alert Report fraud Fraud investigation Fake website checker Scam checker Identity theft protection Consumer protection Chargeback for scam Wire transfer scam Tech support scam Employment scam Rental scam Shopping scam Email scam WhatsApp scam Telegram scam Facebook scam Instagram scam
-
18.07.26 17:12
Malthe Larsen
My experience with OKX has been deeply frustrating. For months, my account withdrawals were restricted with no explanation or resolution. Multiple emails to their support team were ignored, leaving me without answers or reassurance. This complete lack of communication shattered my trust. I ultimately regained access to my funds only through the help of a third-party recovery service, ResQProfirm. What should have been a reliable platform instead made me feel helpless and cut off from my own assets. [email protected], WhatsApp +19852969146, telegram @Resqprofirm
-
18.07.26 17:42
پریا رضایی
OKX has been a major disappointment. My withdrawals were restricted for months, and repeated attempts to contact support went unanswered. This silence destroyed my confidence in the platform. I was only able to recover my funds through a third-party service, ResQProfirm. I trusted OKX for its reliability, but the experience left me feeling trapped and helpless. Timely communication and access to one’s own money should be a basic standard. [email protected], WhatsApp +19852969146, telegram: ResQprofirm
-
18.07.26 17:46
Adem Akışık
I truly didn’t expect such an outstanding outcome. Recovering my $49,360 felt impossible at first, but ResQprofirm’s dedication and persistence made it happen. I hold their team in the highest regard. [email protected], WhatsApp +19852969146
-
18.07.26 23:51
bernalzenaida
WhatsApp https://wa.link/fhle97 Telegram https://msng.link/o?@techcyberforc=tg As cryptocurrency continues to reshape global finance, cybercriminals are finding new ways to exploit investors through scams, hacks, phishing attacks, fake investment platforms, and other forms of digital asset fraud. For many victims, knowing where to turn after a loss can be one of the biggest challenges. Techy Force Cyber Retrieval was founded with one clear mission: to give victims of crypto fraud a fighting chance through professional blockchain investigations and cybersecurity expertise. Our team brings together experienced blockchain analysts, digital forensic specialists, cybersecurity professionals, and legal partners who work collaboratively to investigate cryptocurrency-related crimes. Using advanced blockchain forensic tools and global investigative techniques, we analyze transaction histories, trace digital asset movements where possible, identify valuable investigative leads, and prepare evidence that may assist clients and the appropriate authorities. We believe blockchain should represent transparency, accountability, and trust—not fear. That’s why we’re committed to helping victims understand their options, navigate the investigative process, and take informed action after cryptocurrency fraud. Every case is different, and while no legitimate recovery service can promise a successful recovery, acting quickly and working with experienced professionals can improve the quality of an investigation. At Techy Force Cyber Retrieval, we do more than investigate digital crimes—we advocate for victims, pursue the facts, and help people regain confidence after cryptocurrency fraud. WhatsApp https://wa.link/fhle97 Telegram https://msng.link/o?@techcyberforc=tg Crypto fraud doesn’t have to be the end of the road. It’s where our investigation begins.
-
19.07.26 04:10
Fraddy Pual
I can't thank Fundsretriever enough for everything they did for me. My name is Vanessa Conway, and I'm here to tell you my story of how I recovered money I never thought I'd see again. A few months ago, I put a significant amount of money into what looked like a genuine online investment company. At first, it felt real—they showed me fake profits and convinced me to invest even more. But when I tried to cash out, they went quiet and started asking for extra fees. That's when it hit me—I had been scammed. I was heartbroken, frustrated, and didn't know where to turn. That money was my savings—months of hard work gone. Then I found Fundsretriever online. I reached out, hoping for a miracle. Right away, their team made me feel heard. They were responsive, knowledgeable, and walked me through everything. They didn't just take my case—they took it seriously and kept me in the loop every step of the way. I finally felt like I had real experts fighting for me. And guess what? They actually got my money back. It wasn't instant, and it took teamwork, but it happened. When I saw those funds returned, I cried with joy. I'm sharing this so that anyone else out there who's been scammed knows—don't lose hope. Do your research before investing, and stay far away from platforms that promise too much too fast. And if you've already been stung, don't wait—get professional help immediately. Thank you, Fundsretriever, from the bottom of my heart. You didn't just recover my money—you restored my faith. — Vanessa Conway 📧 [email protected] 📱 WhatsApp: +16035121448 💬 Telegram: @Fundsretriever
-
19.07.26 19:53
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text Number: +1 (336) 390-6684
-
19.07.26 19:53
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text Number: +1 (336) 390-6684
-
19.07.26 19:54
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text Number: +1 (336) 390-6684
-
30.07.26 00:04
Ahmed
A really cool analysis, thank you. I was especially struck by how strictly the order is defined: data processing and formatting come first, with color only at the very end. This really saves you from the typical mistake of "first a pretty palette, then we figure out what the chart is for." And the palette validator with OKLCH + colorblindness check is absolutely fantastic; you almost never see that in regular tools. By the way, while reading about rank-trajectory and the logarithmic scale, I immediately remembered how convenient it is to analyze dynamics on charts in ExpertOption—I've been trading there for quite some time now. When the data is well visualized, decisions are noticeably easier and more relaxed. I also liked the point about "no more than eight colors" and the dual-axis ban. Strict restrictions sometimes actually produce better results than complete freedom. I'll try this approach myself.
-
30.07.26 17:27
wendytaylor015
My name is Wendy Taylor, I'm from Los Angeles, i want to announce to you Viewer how Capital Crypto Recover help me to restore my Lost Bitcoin, I invested with a Crypto broker without proper research to know what I was hoarding my hard-earned money into scammers, i lost access to my crypto wallet or had your funds stolen? Don’t worry Capital Crypto Recover is here to help you recover your cryptocurrency with cutting-edge technical expertise, With years of experience in the crypto world, Capital Crypto Recover employs the best latest tools and ethical hacking techniques to help you recover lost assets, unlock hacked accounts, Whether it’s a forgotten password, Capital Crypto Recover has the expertise to help you get your crypto back. a security company service that has a 100% success rate in the recovery of crypto assets, i lost wallet and hacked accounts. I provided them the information they requested and they began their investigation. To my surprise, Capital Crypto Recover was able to trace and recover my crypto assets successfully within 24hours. Thank you for your service in helping me recover my $647,734 worth of crypto funds and I highly recommend their recovery services, they are reliable and a trusted company to any individuals looking to recover lost money. Contact email [email protected] OR Telegram @Capitalcryptorecover Call/Text Number +1 (336)390-6684 his contact: [email protected] His website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
30.07.26 17:27
wendytaylor015
My name is Wendy Taylor, I'm from Los Angeles, i want to announce to you Viewer how Capital Crypto Recover help me to restore my Lost Bitcoin, I invested with a Crypto broker without proper research to know what I was hoarding my hard-earned money into scammers, i lost access to my crypto wallet or had your funds stolen? Don’t worry Capital Crypto Recover is here to help you recover your cryptocurrency with cutting-edge technical expertise, With years of experience in the crypto world, Capital Crypto Recover employs the best latest tools and ethical hacking techniques to help you recover lost assets, unlock hacked accounts, Whether it’s a forgotten password, Capital Crypto Recover has the expertise to help you get your crypto back. a security company service that has a 100% success rate in the recovery of crypto assets, i lost wallet and hacked accounts. I provided them the information they requested and they began their investigation. To my surprise, Capital Crypto Recover was able to trace and recover my crypto assets successfully within 24hours. Thank you for your service in helping me recover my $647,734 worth of crypto funds and I highly recommend their recovery services, they are reliable and a trusted company to any individuals looking to recover lost money. Contact email [email protected] OR Telegram @Capitalcryptorecover Call/Text Number +1 (336)390-6684 his contact: [email protected] His website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
31.07.26 16:21
rssllhrnsb
I learned an important lesson after investing in what appeared to be a genuine opportunity. Unfortunately, I was unable to access my funds, which reinforced the importance of carrying out thorough due diligence, checking whether an investment is appropriately regulated, and seeking qualified professional advice rather than relying solely on online reviews or testimonials. During the process of addressing my case, I worked with Mrs. Doris Ashley, who communicated clearly, provided regular updates, and handled the matter in a professional manner. According to my experience, I have recovered $50,000 so far, while efforts to resolve the remaining balance are still in progress. If you wish to contact her, the details I used are: Mrs. Tatiana Sorina TEXT : (tatianasorina06 at G.Ma IL ..c 0 m ) Before committing money to any investment, take time to verify the legitimacy of the platform, confirm any relevant regulatory authorisations, and avoid investing more than you can comfortably afford to lose.
-
01.08.26 15:05
keithwilson9899
ETHEREUM RECOVERY ASSISTANCE: CAPITAL CRYPTO RECOVER HELPED ME RECOVER $98,000 WORTH OF LOST ETH In cases of cryptocurrency scams, having accurate information and trusted support is essential. I would like to recommend Capital Crypto Recover Service, a professional team that specializes in assisting individuals with the recovery of lost or stolen Bitcoin and Ethereum (ETH). Their experienced experts are dedicated to helping victims of digital asset fraud by carefully analyzing each case, developing strategic recovery plans, Capital Crypto Recover Service knowledgeable team's primary goals are to satisfy clients and offer significant support and working diligently toward fund retrieval. The team is committed to providing reliable assistance and maintaining a high level of client satisfaction. Based on my assessment, their reputation professionalism and a strong commitment to their clients. If you have experienced a cryptocurrency loss, you can contacting them for further assistance Phone (Call/Text): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
01.08.26 15:05
keithwilson9899
ETHEREUM RECOVERY ASSISTANCE: CAPITAL CRYPTO RECOVER HELPED ME RECOVER $98,000 WORTH OF LOST ETH In cases of cryptocurrency scams, having accurate information and trusted support is essential. I would like to recommend Capital Crypto Recover Service, a professional team that specializes in assisting individuals with the recovery of lost or stolen Bitcoin and Ethereum (ETH). Their experienced experts are dedicated to helping victims of digital asset fraud by carefully analyzing each case, developing strategic recovery plans, Capital Crypto Recover Service knowledgeable team's primary goals are to satisfy clients and offer significant support and working diligently toward fund retrieval. The team is committed to providing reliable assistance and maintaining a high level of client satisfaction. Based on my assessment, their reputation professionalism and a strong commitment to their clients. If you have experienced a cryptocurrency loss, you can contacting them for further assistance Phone (Call/Text): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
03.08.26 20:05
Philip
I was a victim of a crypto theft involving a Pink Drainer, which resulted in the theft of my Wrapped Bitcoin (WBTC) from my Polygon network. The experience was incredibly frustrating and distressing, as I had no idea how to recover my funds. Unfortunately, by the time I noticed the theft, the funds had already been drained to an address that I had no control over, making it seem like an irreversible situation.The attack happened when I clicked on what appeared to be a legitimate link. I didn’t realize at the time that it was a phishing attempt designed to siphon off my private keys and access my wallet. The Pink Drainer, a type of malicious script used by attackers, is specifically designed to exploit such vulnerabilities in crypto wallets. The moment I realized that my WBTC had been drained from my Polygon network. I felt completely helpless, as I didn’t have direct access to the thief's address, and there was no way to reverse the transaction on my own at that point, I started searching for ways to recover my funds, but most resources only offered generic advice that wasn’t practical in this particular case. I quickly realized that if I wanted to have any hope of getting my assets back, I would need professional assistance. After some research, I came across a reliable and trusted recovery team called Aspen Recovery Experts. Their expertise in cryptocurrency recovery, especially in cases like mine, seemed promising.I decided to reach out to Aspen Recovery Experts, and I’m incredibly grateful that I did. Their team of crypto recovery experts was able to help me trace the stolen funds and identify the path the funds took after they left my wallet. Using advanced tools and techniques, they were able to track the transactions on the blockchain, helping me understand where my WBTC had been sent. More importantly, they worked tirelessly to assist me in contacting the necessary parties and even interfaced with blockchain analysts to help facilitate the recovery process.Thanks to their efforts, I was able to successfully recover my stolen funds. The entire process took some time, but the Aspen Recovery Experts dedicated team provided regular updates and kept me informed throughout the process, which gave me a sense of hope and relief during an otherwise stressful time. If you’re ever in a similar situation, I highly recommend reaching out to a trusted recovery team like Aspen Recovery Expertshhh . Their professionalism, knowledge, and expertise were critical in helping me recover my funds and regain control of my crypto assets. Whatsapp : +1 747 231 9036 Telegram : @ prohackerspy Email : [email protected]
-
03.08.26 20:06
Philip
I was a victim of a crypto theft involving a Pink Drainer, which resulted in the theft of my Wrapped Bitcoin (WBTC) from my Polygon network. The experience was incredibly frustrating and distressing, as I had no idea how to recover my funds. Unfortunately, by the time I noticed the theft, the funds had already been drained to an address that I had no control over, making it seem like an irreversible situation.The attack happened when I clicked on what appeared to be a legitimate link. I didn’t realize at the time that it was a phishing attempt designed to siphon off my private keys and access my wallet. The Pink Drainer, a type of malicious script used by attackers, is specifically designed to exploit such vulnerabilities in crypto wallets. The moment I realized that my WBTC had been drained from my Polygon network. I felt completely helpless, as I didn’t have direct access to the thief's address, and there was no way to reverse the transaction on my own at that point, I started searching for ways to recover my funds, but most resources only offered generic advice that wasn’t practical in this particular case. I quickly realized that if I wanted to have any hope of getting my assets back, I would need professional assistance. After some research, I came across a reliable and trusted recovery team called Aspen Recovery Experts. Their expertise in cryptocurrency recovery, especially in cases like mine, seemed promising.I decided to reach out to Aspen Recovery Experts, and I’m incredibly grateful that I did. Their team of crypto recovery experts was able to help me trace the stolen funds and identify the path the funds took after they left my wallet. Using advanced tools and techniques, they were able to track the transactions on the blockchain, helping me understand where my WBTC had been sent. More importantly, they worked tirelessly to assist me in contacting the necessary parties and even interfaced with blockchain analysts to help facilitate the recovery process.Thanks to their efforts, I was able to successfully recover my stolen funds. The entire process took some time, but the Aspen Recovery Experts dedicated team provided regular updates and kept me informed throughout the process, which gave me a sense of hope and relief during an otherwise stressful time. If you’re ever in a similar situation, I highly recommend reaching out to a trusted recovery team like Aspen Recovery Expertshhh . Their professionalism, knowledge, and expertise were critical in helping me recover my funds and regain control of my crypto assets. Whatsapp : +1 747 231 9036 Telegram : @ prohackerspy Email : [email protected]
-
04.08.26 11:05
Kisnoles
Excellent analysis, thank you. I was particularly struck by how rigidly the skill sets the order: data processing and form come first, with color coming last. This really cures the habit of "first a pretty palette, then we'll figure it out." A palette validator using OKLCH + color blindness + WCAG is something most chart generators lack. And regarding the boundaries of competence, it's very clear. Where there's a self-checking loop (color), you delegate freely. Where there's a heuristic (form, data interpretation), you remain the final filter. This is a universal principle, not just for /dataviz. By the way, when you look at trading dashboards (including those of decent platforms like ExpertOption), it's immediately clear who thought about readability and who just threw in rainbow lines. Tools like this skill could greatly improve the quality of analytics. Thanks again for the detailed analysis – saved.
-
04.08.26 12:37
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text Number: +1 (336) 390-6684
-
04.08.26 12:37
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] Call/Text Number: +1 (336) 390-6684
-
06.08.26 08:11
ROMMYHENDERSON344
All you need is to hire an expert to help you accomplish that. If there's any need to spy on your partner's phone. From my experience I lacked evidence to confront my husband on my suspicion on his infidelity, until I came across REALCYBERHACKERS which many commend him of assisting them in their spying mission. So I contacted him and he provided me with access into his phone to view all text messages, call logs, WhatsApp messages and even her location. This evidence helped me move him off my life . I recommend you consult REALCYBERHACKERS AT gmail com or whatsapp +14106350697 if you need access to your partner's phone or any kind of hacking, they carry out all kinds of hacking job
-
06.08.26 13:56
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/Text: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
06.08.26 13:56
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/Text: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
11.08.26 03:43
raymont0714
I recommend a trusted cybersecurity PRO with experience in authorized device access, security testing, and data recovery. SHE work only with the owner's consent and follow all legal and privacy rules on meta data. With permission, this specialist can recover lost files, check device security offline & online, and review texts, call records, or hidden data (spy or lurk around a cheating partner or colleuge). Strict confidentiality agreements protect the information, and client details remain private. The work is careful, efficient, and suited to complex cases. For help securing or recovering information from a phone or another electronic device, use the details below 2 consult [email protected] +44 7476618364 I trust her secrecy a living witness. Her discretion is unmatched, ensuring your privacy is always maintained
-
12.08.26 16:37
rssllhrnsb
I learned an important lesson after investing in what appeared to be a genuine opportunity. Unfortunately, I was unable to access my funds, which reinforced the importance of carrying out thorough due diligence, checking whether an investment is appropriately regulated, and seeking qualified professional advice rather than relying solely on online reviews or testimonials. During the process of addressing my case, I worked with Mrs. Tatiana Sorina , who communicated clearly, provided regular updates, and handled the matter in a professional manner. According to my experience, I have recovered $50,000 so far, while efforts to resolve the remaining balance are still in progress. If you wish to contact her, the details I used are: Mrs. Tatiana Sorina TEXT : (tatianasorina06 at G_Ma IL dot ..c 0 m)…. Before committing money to any investment, take time to verify the legitimacy of the platform, confirm any relevant regulatory authorisations, and avoid investing more than you can comfortably afford to lose.
-
16.08.26 01:44
Matt Kegan
CapitalNode Analytics. They help to investigate and recover stolen digital assets from fake trading platforms. Great firm i must say.
-
18.08.26 04:59
marcushenderson624
Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone WhatsApp/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
18.08.26 04:59
marcushenderson624
Bitcoin Recovery Testimonial After falling victim to a cryptocurrency scam group, I lost $354,000 worth of USDT. I thought all hope was lost from the experience of losing my hard-earned money to scammers. I was devastated and believed there was no way to recover my funds. Fortunately, I started searching for help to recover my stolen funds and I came across a lot of testimonials online about Capital Crypto Recovery, an agent who helps in recovery of lost bitcoin funds, I contacted Capital Crypto Recover Service, and with their expertise, they successfully traced and recovered my stolen assets. Their team was professional, kept me updated throughout the process, and demonstrated a deep understanding of blockchain transactions and recovery protocols. They are trusted and very reliable with a 100% successful rate record Recovery bitcoin, I’m grateful for their help and highly recommend their services to anyone seeking assistance with lost crypto. Contact: [email protected] Phone WhatsApp/Text Number: +1 (336) 390-6684 Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
19.08.26 14:50
BAYER7043
My account was locked, and I couldn’t access my own information until [email protected] +447476618364 whats?up? helped me recover it. This lady hacker was kind and professional, but this experience showed me how risky account lockouts can be. If you’re facing the same problem, do not panic consult her A. S. A. P. Be careful with recovery agents outchea, and research their name, business, and reviews before sharing any details. Never send passwords, security codes, banking information, or copies of your ID unless you’ve confirmed who you’re dealing with. Be wary of promises that sound too good to be true, especially claims that require little information or guarantee instant access with outrageous fees.
-
20.08.26 11:32
michaeldavenport218
I was recently scammed out of $53,000 by a fraudulent Bitcoin investment scheme, which added significant stress to my already difficult health issues, as I was also facing cancer surgery expenses. Desperate to recover my funds, I spent hours researching and consulting other victims, which led me to discover the excellent reputation of Capital Crypto Recover, I came across a Google post It was only after spending many hours researching and asking other victims for advice that I discovered Capital Crypto Recovery’s stellar reputation. I decided to contact them because of their successful recovery record and encouraging client testimonials. I had no idea that this would be the pivotal moment in my fight against cryptocurrency theft. Thanks to their expert team, I was able to recover my lost cryptocurrency back. The process was intricate, but Capital Crypto Recovery's commitment to utilizing the latest technology ensured a successful outcome. I highly recommend their services to anyone who has fallen victim to cryptocurrency fraud. For assistance contact [email protected] and on Telegram OR WhatsApp Number +1 (336)390-6684 via email: [email protected] you can visit his website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
20.08.26 11:32
michaeldavenport218
I was recently scammed out of $53,000 by a fraudulent Bitcoin investment scheme, which added significant stress to my already difficult health issues, as I was also facing cancer surgery expenses. Desperate to recover my funds, I spent hours researching and consulting other victims, which led me to discover the excellent reputation of Capital Crypto Recover, I came across a Google post It was only after spending many hours researching and asking other victims for advice that I discovered Capital Crypto Recovery’s stellar reputation. I decided to contact them because of their successful recovery record and encouraging client testimonials. I had no idea that this would be the pivotal moment in my fight against cryptocurrency theft. Thanks to their expert team, I was able to recover my lost cryptocurrency back. The process was intricate, but Capital Crypto Recovery's commitment to utilizing the latest technology ensured a successful outcome. I highly recommend their services to anyone who has fallen victim to cryptocurrency fraud. For assistance contact [email protected] and on Telegram OR WhatsApp Number +1 (336)390-6684 via email: [email protected] you can visit his website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
23.08.26 20:02
leslieyee
Excellent analysis by /dataviz! I was particularly struck by how strictly the order is defined: "data meaning first, color last" and how the validator actually adjusts the palette according to OKLCH, color blindness, and WCAG standards. These rules greatly improve the quality of charts. By the way, a similar approach to clean and legible charts is highly valued in trading—for example, on the ExpertOption platform, the interface and price visualization are designed to ensure information is read instantly and without unnecessary noise.
-
25.08.26 13:44
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
25.08.26 13:44
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
30.08.26 15:37
[email protected]
Discovering I had been defrauded, I searched online for services claiming to recover stolen ETH. I contacted several companies, but none succeeded. Although some claimed they could trace assets or recover funds, I could not verify their success or recover my money. I later found SYLVESTER BRYANT Recovery through Google. After contacting him, he recovered 450,000,00. He can be reached at [email protected] or WhatsApp at +1 512 577 7957.
-
30.08.26 15:37
[email protected]
Discovering I had been defrauded, I searched online for services claiming to recover stolen ETH. I contacted several companies, but none succeeded. Although some claimed they could trace assets or recover funds, I could not verify their success or recover my money. I later found SYLVESTER BRYANT Recovery through Google. After contacting him, he recovered 450,000,00. He can be reached at [email protected] or WhatsApp at +1 512 577 7957.
-
01.09.26 11:16
lisawerth897
My Experience With Cryptocurrency — CAPITAL CRYPTO RECOVER I lost over $82,000 in Bitcoin after falling victim to a fraudulent online investment scheme. After realizing I had been scammed, I spent considerable time researching possible ways to recover my funds. During my research, I came across CAPITAL CRYPTO RECOVER and decided to contact them after reading their reported success record and encouraging client positive reviews and testimonials. Their team guided me through the recovery process, and I was grateful to successfully recover my lost cryptocurrency. The experience was challenging, but I appreciated the assistance and support I received throughout the process. I’m sharing my experience in the hope that it may encourage other victims to carefully research their options and verify any recovery service before proceeding. I will forever be thankful to you CAPITAL CRYPTO RECOVER 📧 [email protected] 🌐 Website: recovercapital.wixsite.com/capital-crypto-rec-1 📧 [email protected] 📞 Call/WhatsApp: +1 (336) 390-6684
-
01.09.26 11:16
lisawerth897
My Experience With Cryptocurrency — CAPITAL CRYPTO RECOVER I lost over $82,000 in Bitcoin after falling victim to a fraudulent online investment scheme. After realizing I had been scammed, I spent considerable time researching possible ways to recover my funds. During my research, I came across CAPITAL CRYPTO RECOVER and decided to contact them after reading their reported success record and encouraging client positive reviews and testimonials. Their team guided me through the recovery process, and I was grateful to successfully recover my lost cryptocurrency. The experience was challenging, but I appreciated the assistance and support I received throughout the process. I’m sharing my experience in the hope that it may encourage other victims to carefully research their options and verify any recovery service before proceeding. I will forever be thankful to you CAPITAL CRYPTO RECOVER 📧 [email protected] 🌐 Website: recovercapital.wixsite.com/capital-crypto-rec-1 📧 [email protected] 📞 Call/WhatsApp: +1 (336) 390-6684
-
01.09.26 17:34
Garry42
Really crazy world. These fraudsters go at any length to steal your hard earned funds. I have been a victim of a bitcoin scam about 7 months back. A Con artist gained access to my cashapp account through a phishing scam. They stole $409,000. I was really devastated. I did everything to get back my funds by contacting the FBI but they claimed there was nothing they could do. A friend told me about a recovery expert. He helps fight against various phishing and investment scams and they were able to help trace and recover my funds even though it took over 2 days. you can reach out to him through recoverydarek@gmail. com . I can guarantee his services are still active.
-
01.09.26 17:34
Garry42
Really crazy world. These fraudsters go at any length to steal your hard earned funds. I have been a victim of a bitcoin scam about 7 months back. A Con artist gained access to my cashapp account through a phishing scam. They stole $409,000. I was really devastated. I did everything to get back my funds by contacting the FBI but they claimed there was nothing they could do. A friend told me about a recovery expert. He helps fight against various phishing and investment scams and they were able to help trace and recover my funds even though it took over 2 days. you can reach out to him through recoverydarek@gmail. com . I can guarantee his services are still active.
-
02.09.26 03:43
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
02.09.26 03:43
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
04.09.26 21:51
Kovengray
Recovering your lost investment funds as the case might be, is not what you can do alone, you’d require the service of a trained recovery specialist. A recovery specialist is a person or a group of people who are well equipped to work around the brokerage network. They have vast knowledge about the whole network and have the right software and private keys to follow any transaction. I was ripped off trading online to an investment broker, good thing I got every penny back through the help of Gavin ray he’s a genius Contact : Gavinray78 at gmail com or WhatsApp +1 352 322 2096 It is also important to be patient and really calm during the process.
-
05.09.26 21:38
[email protected]
I invested 45,000 Euro, and later aggreviated to 198,000 Euro. I requested to place Withdrawal of my funds to be paid to my Bank account. But nothing happened I was subjected to pay more fee until I can get my funds on my account, i got in touch with Theodore ryan here on this platform who had helped a lot of people, I followed all instructions and he legally got back my withheld funds, I got threatened by the company that if I don’t pay they will get my account frozen, all thanks to Theodore ryan and I highly recommend him to anyone dealing with an unregulated broker company… contact him on his Gmail - theodoreryan318@ gmail . com
-
09.09.26 21:31
lisawerth897
My Experience With Cryptocurrency — CAPITAL CRYPTO RECOVER I lost over $82,000 in Bitcoin after falling victim to a fraudulent online investment scheme. After realizing I had been scammed, I spent considerable time researching possible ways to recover my funds. During my research, I came across CAPITAL CRYPTO RECOVER and decided to contact them after reading their reported success record and encouraging client positive reviews and testimonials. Their team guided me through the recovery process, and I was grateful to successfully recover my lost cryptocurrency. The experience was challenging, but I appreciated the assistance and support I received throughout the process. I’m sharing my experience in the hope that it may encourage other victims to carefully research their options and verify any recovery service before proceeding. I will forever be thankful to you CAPITAL CRYPTO RECOVER 📧 [email protected] 🌐 Website: recovercapital.wixsite.com/capital-crypto-rec-1 📧 [email protected] 📞 Call/WhatsApp: +1 (336) 390-6684
-
09.09.26 21:31
lisawerth897
My Experience With Cryptocurrency — CAPITAL CRYPTO RECOVER I lost over $82,000 in Bitcoin after falling victim to a fraudulent online investment scheme. After realizing I had been scammed, I spent considerable time researching possible ways to recover my funds. During my research, I came across CAPITAL CRYPTO RECOVER and decided to contact them after reading their reported success record and encouraging client positive reviews and testimonials. Their team guided me through the recovery process, and I was grateful to successfully recover my lost cryptocurrency. The experience was challenging, but I appreciated the assistance and support I received throughout the process. I’m sharing my experience in the hope that it may encourage other victims to carefully research their options and verify any recovery service before proceeding. I will forever be thankful to you CAPITAL CRYPTO RECOVER 📧 [email protected] 🌐 Website: recovercapital.wixsite.com/capital-crypto-rec-1 📧 [email protected] 📞 Call/WhatsApp: +1 (336) 390-6684
-
09.09.26 23:22
Fraddy Pual
Hearing that these individuals are focusing on other people makes me very sad. I had a similar situation with them and lost a lot of money, but after learning about (Cruxcipherteam @ proton DoT me), whataqq:+168160-15021, telegram: @Cruxcipherteam I was able to get my money back. It's critical that we all be watchful and keep reporting these occurrences.
-
11.09.26 03:23
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
11.09.26 03:23
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
15.09.26 03:35
elioduncan
I fell victim to a freelance web development contract scam that ultimately cost me CAD 4,000. It all started when I came across a job posting on Indeed Canada, a popular online job portal. The position seemed perfect: a freelance web developer role for an established company looking for someone to design and build a fully functional e-commerce website. The job promised a high payment of CAD 20,000 upon successful completion, which sounded like a great opportunity for me to gain experience and earn decent pay.The employer, who introduced himself as a project manager from a "well-known" tech company, was very persuasive and professional in our initial communication. He explained that the project involved developing a user-friendly, responsive online store for their client and that they had a strict timeline to meet. He assured me that the payment would be made promptly after completing the tasks. However, before starting the work, he told me that I would need to pay an upfront fee of CAD 4,000 to cover certain software tools and licensing fees required for the project. This was supposedly a part of their company policy for freelance contractors, ensuring access to their premium resources. The idea of working on a professional project, coupled with the promise of a substantial payout, convinced me to pay the upfront fee.As soon as I made the payment, the project manager became increasingly difficult to reach. He initially responded to my emails and provided some vague instructions on what the project would entail, but as time went on, communication slowed to a complete halt. I never received the required tools or any proper project details. My emails went unanswered, and any attempts to contact the company were met with silence. After waiting for weeks, I realized that I had been scammed.Desperate to recover my money, I turned to TechY Force Cyber Retrieval, a service that specializes in helping victims of online scams. They helped me track the payment and took legal steps to pursue the fraudsters. Through their guidance, I was able to recover the full CAD 4,000. TechY Force Cyber Retrieval worked with my bank to reverse the transaction and liaised with the authorities to trace the scammer's details.This was a hard lesson, and I now know to be highly cautious about online job offers that require upfront payments. It is crucial to research companies thoroughly and avoid any job that seems too good to be true, especially when it involves paying money upfront. WhatsApp https://wa.link/2x6ktp Mail. [email protected]
-
15.09.26 15:45
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
15.09.26 15:45
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
05.10.26 21:26
[email protected]
I Invested 45,000 Euro, and later aggreviated to 198,000 Euro. I requested to place Withdrawal of my funds to be paid to my Bank account. But nothing happened I was subjected to pay more fee until I can get my funds on my account, i got in touch with Theodore ryan here on this platform who had helped a lot of people, I followed all instructions and he legally got back my withheld funds, I got threatened by the company that if I don’t pay they will get my account frozen, all thanks to Theodore ryan and I highly recommend him to anyone dealing with an unregulated broker company… contact him on his Gmail - theodoreryan318@ gmail . com
-
06.10.26 06:13
melissathomason573
Professional Crypto Recovery Testimonial After losing a significant amount of cryptocurrency through a scam on Twitter, I was unsure whether my Bitcoin funds could ever be recovered. The experience was extremely stressful, particularly because I encountered several individuals and services claiming they could help recover my assets, only to discover that some were fraudulent themselves. A friend eventually referred me to CAPITAL CRYPTO RECOVER. Their team handled my case professionally, discreetly, and efficiently. With their assistance, I successfully recovered approximately $130,000 worth of USDT in just over a week. My experience taught me the importance of being extremely cautious when seeking cryptocurrency recovery services, as scammers often target victims a second time. I am grateful to CAPITAL CRYPTO RECOVER for the assistance I received and wanted to share my experience with others who may be facing a similar situation. If you’ve been compromised, don’t lose hope—and be careful of fraudsters posing as saviors. CAPITAL CRYPTO RECOVER are real professionals in crypto recovery. I am a living proof of their effectiveness, you can reach them through Phone (Call/WhatsApp): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
06.10.26 06:13
melissathomason573
Professional Crypto Recovery Testimonial After losing a significant amount of cryptocurrency through a scam on Twitter, I was unsure whether my Bitcoin funds could ever be recovered. The experience was extremely stressful, particularly because I encountered several individuals and services claiming they could help recover my assets, only to discover that some were fraudulent themselves. A friend eventually referred me to CAPITAL CRYPTO RECOVER. Their team handled my case professionally, discreetly, and efficiently. With their assistance, I successfully recovered approximately $130,000 worth of USDT in just over a week. My experience taught me the importance of being extremely cautious when seeking cryptocurrency recovery services, as scammers often target victims a second time. I am grateful to CAPITAL CRYPTO RECOVER for the assistance I received and wanted to share my experience with others who may be facing a similar situation. If you’ve been compromised, don’t lose hope—and be careful of fraudsters posing as saviors. CAPITAL CRYPTO RECOVER are real professionals in crypto recovery. I am a living proof of their effectiveness, you can reach them through Phone (Call/WhatsApp): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
06.10.26 17:44
Darrell1717
I fell victim of a scam I invested to total of 5thousand into. I know it could be risky but after I trusted this man I was dating online he lured me into investing with him. I know hard I worked of my money so I wanted every way possible to get it back. I really appreciate Blockrecoveryexpert for successfully recovered my funds form this scammer. This is all I can say. You can reachout if you ever need help contact blockrecoveryexpert (at) gmail (dot)com
-
11.10.26 03:04
Patricia014
Most recovery companies will take your money again. I personally think they’re the same set of people running all of them. It’s a sad, unending cycle, and so many people fall for cloned websites. The only recovery agent I know is Agent Rottgen Raphael. I’ve visited their physical company address, and they’ll tell you if they can handle your case instead of taking upfront payment when they know they can’t help. I highly appreciate their support in recovering my money from the scammers’ wallet within 24 hours. RottgenRaphael@Gmail. com
-
12.10.26 19:34
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
12.10.26 19:34
lydiassmith567
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
13.10.26 14:14
lisalisa22
Legitimate Cryptocurrency Recovery Services: Consult ZEUS CRYPTO RECOVERY SERVICES Zeus Crypto Recovery Services supports individuals exploring options after cryptocurrency losses from fraud, unauthorised transfers, or wallet access issues. Services may include blockchain analysis, wallet tracing, and evidence gathering for investigations. Each case is reviewed. Zeus Crypto Recovery Services is a trusted company with the highest success rate among cryptocurrency recovery services in 2025/2026 and is highly recommended if you have been scammed. ZEUS CRYPTO RECOVERY SERVICES handles case such as Scammed USDT Lost BTC, Coinbase Wallet Forgotten password, ETH and coins, etc. Anything that has to do with cryptocurrency or lost digital assets, ZEUS can help you Do you Need Help? Website https://zeusrecoveryservices.com Mailbox [email protected] WhatsApp number +447798816938 Telegram https://t.me/Zeuscryptorecoveryservices
-
13.10.26 14:14
lisalisa22
Legitimate Cryptocurrency Recovery Services: Consult ZEUS CRYPTO RECOVERY SERVICES Zeus Crypto Recovery Services supports individuals exploring options after cryptocurrency losses from fraud, unauthorised transfers, or wallet access issues. Services may include blockchain analysis, wallet tracing, and evidence gathering for investigations. Each case is reviewed. Zeus Crypto Recovery Services is a trusted company with the highest success rate among cryptocurrency recovery services in 2025/2026 and is highly recommended if you have been scammed. ZEUS CRYPTO RECOVERY SERVICES handles case such as Scammed USDT Lost BTC, Coinbase Wallet Forgotten password, ETH and coins, etc. Anything that has to do with cryptocurrency or lost digital assets, ZEUS can help you Do you Need Help? Website https://zeusrecoveryservices.com Mailbox [email protected] WhatsApp number +447798816938 Telegram https://t.me/Zeuscryptorecoveryservices
-
17.10.26 12:58
melissathomason573
Professional Crypto Recovery Testimonial After losing a significant amount of cryptocurrency through a scam on Twitter, I was unsure whether my Bitcoin funds could ever be recovered. The experience was extremely stressful, particularly because I encountered several individuals and services claiming they could help recover my assets, only to discover that some were fraudulent themselves. A friend eventually referred me to CAPITAL CRYPTO RECOVER. Their team handled my case professionally, discreetly, and efficiently. With their assistance, I successfully recovered approximately $130,000 worth of USDT in just over a week. My experience taught me the importance of being extremely cautious when seeking cryptocurrency recovery services, as scammers often target victims a second time. I am grateful to CAPITAL CRYPTO RECOVER for the assistance I received and wanted to share my experience with others who may be facing a similar situation. If you’ve been compromised, don’t lose hope—and be careful of fraudsters posing as saviors. CAPITAL CRYPTO RECOVER are real professionals in crypto recovery. I am a living proof of their effectiveness, you can reach them through Phone (Call/WhatsApp): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
17.10.26 12:58
melissathomason573
Professional Crypto Recovery Testimonial After losing a significant amount of cryptocurrency through a scam on Twitter, I was unsure whether my Bitcoin funds could ever be recovered. The experience was extremely stressful, particularly because I encountered several individuals and services claiming they could help recover my assets, only to discover that some were fraudulent themselves. A friend eventually referred me to CAPITAL CRYPTO RECOVER. Their team handled my case professionally, discreetly, and efficiently. With their assistance, I successfully recovered approximately $130,000 worth of USDT in just over a week. My experience taught me the importance of being extremely cautious when seeking cryptocurrency recovery services, as scammers often target victims a second time. I am grateful to CAPITAL CRYPTO RECOVER for the assistance I received and wanted to share my experience with others who may be facing a similar situation. If you’ve been compromised, don’t lose hope—and be careful of fraudsters posing as saviors. CAPITAL CRYPTO RECOVER are real professionals in crypto recovery. I am a living proof of their effectiveness, you can reach them through Phone (Call/WhatsApp): +1 (336) 390-6684 Email: [email protected] Alternate Email: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
24.10.26 03:16
lxbfYeaa
e
-
24.10.26 03:16
lxbfYeaa
e
-
24.10.26 03:16
lxbfYeaa
e
-
24.10.26 03:16
lxbfYeaa
e
-
24.10.26 03:23
lxbfYeaa
e
-
24.10.26 03:23
lxbfYeaa
e
-
24.10.26 03:24
lxbfYeaa
e
-
25.10.26 14:35
lydiasmitth456
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
25.10.26 14:35
lydiasmitth456
HIRE A HACKER YOUR STOLEN CRYPTO RECOVERY / BTC / USDT / ETH WITH THE HELP OF CAPITAL CRYPTO RECOVER. I want to share my experience publicly regarding cryptocurrency wallet recovery, I highly recommend you to contact CAPITAL CRYPTO RECOVER, a professional private investigator in the Bitcoin world. They have a certified expert security team specializing in Bitcoin Recovery Services and have helped many people worldwide recover their lost funds. My wife and I were defrauded by an online manipulator posing as an experienced crypto investment professional. We lost $9.2 Million Stolen BTC in cryptocurrency and were left feeling homeless. After spending hours searching for a reliable crypto recovery service, I discovered CAPITAL CRYPTO RECOVER online. By patiently explaining my situation to their team, I was able to recover all my funds. Remarkably, my money was returned to my wallet in less than 24 hours. I am extremely grateful to CAPITAL CRYPTO RECOVER for their excellent assistance—they truly were a godsend in my difficult situation. If you have fallen victim to a cryptocurrency scam, you can reach CAPITAL CRYPTO RECOVER through the following channels Email: [email protected] OR Call/WhatsApp: +1 (336) 390-6684 Contact: [email protected] Website: https://recovercapital.wixsite.com/capital-crypto-rec-1
-
27.10.26 01:58
KATHLEEN
Cyberspacetechexpert focuses on recovering stolen cryptocurrency, particularly USDT and bitcoin. She is well-known for helping victims of digital asset theft. Her reputation arises from successful recoveries that have allowed many to regain their lost funds. I witnessed this when $74,000 was taken from me in January. Thanks to the best female hacker Gracias, I recovered the entire amount in just 14 hours. Her prompt response and effective methods relieved my financial burden. Her commitment to helping others is evident. She is always available to offer solutions to those facing similar problems. For assistance, she can be reached via email Cyberspacetechexpert@gmail .com. or speak to her on What App at + 44 7476618364.
-
27.10.26 02:22
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
27.10.26 02:22
kimberlyhebertt6877
I invested in bitcoin trading After losing $78.4 USDT) linked to a romance fraud scam worth of cryptocurrency through an online investment platform and later discovered it was a scam. After extensive research for recovery options, I contacted CAPITAL CRYPTO RECOVER based on positive client reviews and recommendations. Their professional security team guided me through the recovery process using advanced technology, and I was able to recover my lost cryptocurrency successfully. I am truly grateful for their support and assistance during such a difficult experience. I will advise you to contact CAPITAL CRYPTO RECOVER helped me recover my funds. For anyone facing similar issues, Website: https://recovercapital.wixsite.com/capital-crypto-rec-1 Email: [email protected] Telegram: @Capitalcryptorecover Contact: [email protected] WhatsApp/Text Number: +1 (336) 390-6684
-
27.10.26 02:25
KATHLEEN
Cyberspacetechexpert focuses on recovering stolen cryptocurrency, particularly USDT and bitcoin. She is well-known for helping victims of digital asset theft. Her reputation arises from successful recoveries that have allowed many to regain their lost funds. I witnessed this when $74,000 was taken from me in January. Thanks to the best female hacker Gracias, I recovered the entire amount in just 14 hours. Her prompt response and effective methods relieved my financial burden. Her commitment to helping others is evident. She is always available to offer solutions to those facing similar problems. For assistance, she can be reached via email Cyberspacetechexpert@gmail .com. or speak to her on What App at + 44 7476618364.
-
27.10.26 23:04
kevinlong
Telegram : ( @Drkevinlong) Whatsapp : +237 6 71 10 09 15 SIGNAL : DrLong260.31 Zangi number: 9340169421 Telegram : ( @Drkevinlong) Buy undetectable counterfeit money online counterfeit money sale https://premiumcounterfeitmoney.com/counterfeit-money/ buy counterfeit dollar <a href="https://premiumcounterfeitmoney.com/counterfeit-money/">buy counterfeit dollar</a> <a href="https://premiumcounterfeitmoney.com/where-to-get-counterfeit-money-in-australia/">undetectable counterfeit money for sale</a> <a href="https://premiumcounterfeitmoney.com/buy-counterfeit-australian-banknotes/ purchase counterfeit money where can i buy counterfeit money buy counterfeit money online Undetectable Counterfeit Bank Notes Buy Counterfeit Banknotes Online Undetectable counterfeit Bank Notes Buy Counterfeit Banknotes Online Premium Monetary Coltd has become extremely popular nowadays due to our obvious advantages. First of all, we are cheaper than others, and you do not need to pay any additional custom fee Counterfeit Banknotes. best place to buy counterfeit money online Secondly, realistic prop notes it’s a great way to finally being able to afford those expensive items you would have never thought of buying before. And last but not least, you can order Counterfeit money from the comfort of your home. Just choose the currency you need, mention the required amount, and proceed to checkout. You will get your bills in a matter of days and will be able to start living the life the way you want. Sounds like a dream? Here at Premium Counterfeit Money, we will show you that it is a reality. Undetectable counterfeit Bank Notes | Buy Counterfeit Banknotes Online With more than 8 years of experience behind our belts, we provide our customers with Counterfeit money for sale of unparalleled quality that unlikely to be detected as not genuine. We use the latest printing techniques to make sure the bills you order from us will pass any security checks. When shopping with us, you can rest easy knowing that you will get the best forged bills on the market that have all the security features, including watermarks, security threads, and foil elements. Undetectable counterfeit Bank Notes | Buy Counterfeit Banknotes Online Buy undetected counterfeit bank notes at the best prices worldwide from our online store. This blog will look at the different aspects of using our online store and show why you should buy undetected counterfeit bank notes from us. Undetectable counterfeit Bank Notes | Buy Counterfeit Banknotes Online Buy high quality undetectable counterfeit Banknotes Buy Fake Euros Online ,Experience the pinnacle of authenticity with your premier destination to buy authentic fake Euro bills online in Europe. As the best supplier of fake money Euro located in California, USA, with over 15 years of experience and a workforce of over 410 employees, we are dedicated to providing the highest quality counterfeit Euro to European countries. Last year alone, we printed billions of fake Euro using the most advanced and realistic chemicals and technology to ensure our counterfeit Euro is highly scannable and undetectable. Our Euro bills pass all UV tests and are perfect for various purposes – from financial transactions to movies, clubs, restaurants, educational activities, and more. We deliver to countries that use Euro bills, including Germany, France, Italy, Spain, Netherlands, Belgium, Austria, and Portugal. With a track record of excellence and great customer service. Buy %100 Realistic Counterfeit €5, €10, €20, €50, €100 Bills Online | Most Reliable Supplier of Euro Counterfeit Bills. https://premiumcounterfeitmoney.com/ <a href="https://premiumcounterfeitmoney.com/product/counterfeit-canadian-dollars/">supreme quality undetectable counterfeit money</a> Buy Counterfeit Euro Online Fake Euro Money For Sale https://premiumcounterfeitmoney.com/product/counterfeit-us-dollars/ Best counterfeit money for sale Are you a avid collector of numismatic paper money? Our digital exchange is your premier destination for purchasing prized pieces from around the planet. We feature a diverse selection of rare currency, spanning from <a href="https://premiumcounterfeitmoney.com/buy-fake-euros-online/"> buy fake currency online</a> <a href="https://premiumcounterfeitmoney.com/counterfeit-australian-dollars/">where can i buy counterfeit money</a> <a href="https://premiumcounterfeitmoney.com/buy-undetectable-counterfeit-money-in-israel/">purchase counterfeit money</a> <a href="https://premiumcounterfeitmoney.com/product/counterfeit-us-dollars/">buy counterfeit money online</a>
Для участия в Чате вам необходим бесплатный аккаунт pro-blockchain.com
Войти
Регистрация
Похожие новости
Подпишись на telegram канал и узнавай первым обо всех новостях