How to track new pools on Raydium with Shyft gRPC

Real-Time Raydium Pool Tracking: Never Miss a New Pool Launch Again

Raydium, an Automated Market Maker (AMM) on the Solana blockchain, facilitates decentralized token swaps by leveraging liquidity pools. These pools act as virtual reservoirs where users can directly exchange tokens, bypassing the need for a central order book. For users who are actively swapping and trading tokens, or for Trading Bots or building real-time applications on Raydium, it is crucial to stay updated on platform’s events, such as the creation of new Pools, current state of Pool, or tokens swapped. Shyft’s gRPC services can address this need by providing real-time streaming capabilities, ensuring users are immediately informed of any changes or updates involving any pools on Raydium for more efficient monitoring and timing swaps strategically.

In this article, we will explore how we can track newly created Pools on Raydium in real-time using Shyft’s gRPC services.

Looking for a full working example to following along with? Please refer to this replit project we have created here. Make sure you get your gRPC URL and access token from Shyft Discord here, and add it to the secrets section before executing.

Before Getting Started

To get started, we will need a few things.

Authentication: Your Shyft API Key, gRPC endpoint and gRPC token

You can get your own Shyft API Key(an auth parameter used by Shyft) from the Shyft website. You can also find your region-specific gRPC endpoint and access token on your Shyft Dashboard.

A server-side backend (like NodeJS) to receive gRPC data

As gRPC services are unsupported in web-browsers, you would need a backend application to receive gRPC data. For this example, we have used NodeJS, but any other backend server side languages such as C#, Go, Java, Kotlin, Python or PHP can also be used.

Introduction

Shyft’s geyser-powered gRPC provides streaming services for account updates, transactions, blocks and slot updates, as soon as they happen on-chain. Our plan involves integrating Shyft’s gRPC service to establish a transaction streaming pipeline for Raydium Liquidity Pool v4.

Once we get this data in our backend, we can quickly analyze it to identify transactions that create new Pools on Raydium. By using specific encoded values and empty byte arrays for filtering, we can efficiently retrieve only the important data we need. This way, we focus on liquidity pools related to SOL and Serum markets while still capturing active trading events.

Initialization — gRPC access and Client

Shyft’s gRPC access tokens are available in their discord server. Please refer to the ‘Before Getting Started’ section for more details.

Once we obtain the access token and gRPC URL, we proceed by importing the client class from the Yellow Stone services. Initializing a new client requires two parameters: the gRPC URL and the access token. The client can be initialized as follows:

Shyft offers Dragonmouth gRPC nodes, originally developed by Triton One as part of their Yellowstone project. These nodes enable real-time data streaming from the Solana network.

To connect to these nodes using Node.js, you’ll need the @triton-one/yellowstone-grpc SDK. After installation, you can initialize a client object. This involves:

1. Importing the Client Class: Import the Client class from the @triton-one/yellowstone-grpc SDK.
2. Obtaining Credentials: Acquire your access token and the gRPC URL from Shyft.
3. Initializing the Client: Use the Client class to create a new client object. Provide both the access token and the gRPC URL as parameters during initialization.

The client can be initialized in the following manner.

import Client from "@triton-one/yellowstone-grpc";

const client = new Client(
  "https://grpc.us.shyft.to", //Your Region specific Shyft gRPC URL
  "hbdj-asjnf-access-token-asdh", //Shyft gRPC Access Token
  undefined,
);

For a more detailed guide on how to setup the client and streaming transactions using gRPC, feel free to checkout this article on Streaming real-time Solana Transactions.

Enabling Live Updates — Requesting and Receiving the Stream

Shyft’s gRPC service leverages subscription streams to facilitate real-time updates from the Solana blockchain. These streams act as persistent connections, allowing your application to continuously receive updates as events occur on-chain. To set up a new subscription stream on the client created in the previous step, use the client.subscribe() method.

const stream = await client.subscribe();

With the subscription stream established, it’s time to specify what data you want to receive. Shyft’s gRPC interface offers a variety of updates, including account changes, transactions, new blocks, and even slot updates. To avoid information overload, you can specify what you actually need through subscribe requests.

These subscribe requests come in different format, each with its own set of parameters and they look somewhat like this

import { CommitmentLevel } from "@triton-one/yellowstone-grpc";

const req: SubscribeRequest = {
  accounts: {},
  slots: {},
  transactions: {},
  transactionsStatus: {},
  entry: {},
  blocks: {},
  blocksMeta: {},
  accountsDataSlice: [],
  ping: undefined,
  commitment: CommitmentLevel.CONFIRMED,
};

Most of the request parameters are self-explanatory, working exactly with what they are named,

• ‘accounts’: You can subscribe to specific accounts (e.g., SOL-USDC OpenBook) by specifying this parameter and receive updates based on commitment levels (processed, confirmed, finalized).
• ‘accountDataSlice’: This field helps you to filter your gRPC stream, so that you receive only the relevant portion of streamed data. For example you are streaming accounts, for which the data size is 200bytes, but you only need 40 bytes after a certain offset. This field can help you filter those 40 bytes for every update in the stream.
• ‘transactions’ & ‘transactionsStatus’: You can receive updates on all transactions or filter them based on specific criteria (vote/failed transactions, including/excluding accounts). Programs can also be monitored using this.
• ‘slots’, ‘blocks’ & ‘blocksMeta’ : Stay informed about new blocks and slots being produced on the blockchain.
• ‘commitment’: This specifies the commit level for any update, either processed, confirmed or finalized.

In this tutorial we will be making use of AccountDataSlice. Here is a sample subscription request to receive transactions for Raydium:

const request: SubscribeRequest = {
  "slots": {},
  "accounts": {
    "raydium": {
      "account": [],
      "filters": [
        {
          "memcmp": {
            "offset": LIQUIDITY_STATE_LAYOUT_V4.offsetOf('quoteMint').toString(), // Filter for only tokens paired with SOL
            "base58": "So11111111111111111111111111111111111111112"
          }
        },
        {
          "memcmp": {
            "offset": LIQUIDITY_STATE_LAYOUT_V4.offsetOf('marketProgramId').toString(), // Filter for only Raydium markets that contain references to Serum
            "base58": "srmqPvymJeFKQ4zGQed1GFppgkRHL9kaELCbyksJtPX"
          }
        },
        {
          "memcmp": {
            "offset": LIQUIDITY_STATE_LAYOUT_V4.offsetOf('swapQuoteInAmount').toString(), // Hack to filter for only new tokens. There is probably a better way to do this
            "bytes": Uint8Array.from([0])
          }
         },
        {
          "memcmp": {
            "offset": LIQUIDITY_STATE_LAYOUT_V4.offsetOf('swapBaseOutAmount').toString(), // Hack to filter for only new tokens. There is probably a better way to do this
            "bytes": Uint8Array.from([0])
          }
        }
      ],
      "owner": ["675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8"] // raydium program id to subscribe to
    }
  },
  "transactions": {},
  "blocks": {},
  "blocksMeta": {
    "block": []
  },
  "accountsDataSlice": [],
  "commitment": CommitmentLevel.PROCESSED, // Subscribe to processed blocks for the fastest updates
  entry: {},
  transactionsStatus: {}
}

In this code snippet, we are configuring a subscription to a Solana account to monitor Raydium liquidity pools with specific filters. Here’s a detailed breakdown of the configuration:

1. slots: This parameter is an empty object, indicating that no specific slot data is being subscribed to.
2. accounts: This section configures the subscription to specific Solana accounts. In this case, we are focusing on the Raydium program:

I. raydium: This is the name of the account category for which we are setting up filters.

a. account: An empty array, meaning there are no specific accounts being subscribed to directly.

b. filters: This is a set of criteria to narrow down the accounts that we are interested in:

b.i. : memcmp: This filter is used to match data at specific byte offsets within the account data. The offset specifies the byte position to check, and base58 or bytes specifies the data to match, it tells the system where to look within that binary structure to find specific information. i.e. "offset":LIQUIDITY_STATE_LAYOUT_V4.offsetOf('quoteMint').toString();

c. quoteMint: Filters accounts where the quoteMint field matches a specific base58-encoded mint address (So11111111111111111111111111111111111111112). This is used to focus on tokens paired with SOL. By matching it with a base58 encoded value (like "So11111111111111111111111111111111111111112"), you’re filtering for liquidity pools that specifically pair with SOL. This ensures that you only receive data relevant to trades or liquidity involving SOL, making your analysis more focused and efficient.

d. marketProgramId: Filters accounts where the marketProgramId field matches the base58-encoded Serum program ID (srmqPvymJeFKQ4zGQed1GFppgkRHL9kaELCbyksJtPX). The base58 value (e.g., "srmqPvymJeFKQ4zGQed1GFppgkRHL9kaELCbyksJtPX") corresponds to the Serum market program. By filtering for this value, you can ensure that you’re only monitoring liquidity pools that interact with the Serum decentralized exchange, which is critical for tracking relevant trading activity and price movements.

e. swapQuoteInAmount and swapBaseOutAmount: These fields are used as a crude filter to identify new tokens by checking for accounts where these values are zero. This approach is a workaround and may not be the most efficient method for filtering new tokens.

f. owner: Specifies the owner of the accounts we are interested in. In this case, we are targeting accounts owned by the Raydium program ID (675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8).

3. transactions, blocks, blocksMeta, accountsDataSlice, entry, transactionsStatus: These are set as empty objects or arrays, indicating no specific configurations or subscriptions for transaction data, blocks, or other parameters in this request.

4. commitment: This field defines the level of confirmation for the subscribed data:

I. CommitmentLevel.PROCESSED: This level ensures that you receive updates as soon as they are processed by the Solana network, offering the fastest updates but potentially less confirmation.

With the stream successfully set up, it is necessary to have a function to process the transactions received for further actions. The stream.on('data', callbackFunc()) method handles the stream, as illustrated below.

//callback function that handles the stream
 stream.on("data", async (data) => {
    try{
      if (data.account != undefined) {
       const info = await tOutPut(data);
       console.log(
        new Date(),
        ":",
        `New transaction https://translator.shyft.to/tx/${info.signature} \n`,
        JSON.stringify(info, null, 2) + "\n",
      );
    }
}catch(error){
  if(error){
    console.log(error)
  }
}
});

Precautions — A reconnect mechanism

While gRPC streaming offers a reliable way to receive real-time data, unexpected network issues can still disrupt the connection. To ensure a seamless experience, you can implement a reconnect mechanism in your application. Here is an example mechanism will automatically attempt to re-establish the gRPC stream if it gets disconnected.

async function subscribeCommand(client: Client, args: SubscribeRequest) {
  while (true) {
    try {
      await handleStream(client, args); //function which handles the stream
    } catch (error) {
	    //iff the stream disconnects due to any error, this will wait for a second and restart the stream
      console.error("Stream error, restarting in 1 second...", error);
      await new Promise((resolve) => setTimeout(resolve, 1000));
    }
  }
}

Receiving Transactions and Identifying new Pools

Thus we have seen Shyft’s gRPC services, you can effortlessly stream Solana transactions in real-time. Here is a raw transaction received from the gRPC:

2024-09-07T08:04:52.816Z : New transaction https://translator.shyft.to/tx/2y9VYJ944mZM2LUms5TSNH6cfSjqQMH1sde62bb2KCG6s7xXEDYV6wgCoQAa8ZSUF8AAdM19rvJ6UAnmsqYxPttM
 {
  "filters": [
    "raydium"
  ],
  "account": {
    "account": {
      "pubkey": {
        "type": "Buffer",
        "data": [
          127,
          210,
          158,
          190,
          141,
          181,
          87,
          182,
          122,
          33,
          191,
          96,
          181,
          133,
          115,
          87,
          254,
          58,
          71,
          41,
          63,
          49,
          147,
          127,
          94,
          42,
          248,
          21,
          104,
          234,
          112,
          121...
        ]
      },
      "lamports": "6124800",
      "owner": {
        "type": "Buffer",
        "data": [
          75,
          217,
          73,
          196,
          54,
          2,
          195,
          63,
          32,
          119,
          144,
          237,
          22,
          163,
          82,
          76,
          161,
          185,
          151,
          92,
          241,
          33,
          162,
          169,
          12,
          255,
          236,
          125,
          248,
          182,
          138,
          205
        ]
      },
      "executable": false,
      "rentEpoch": "18446744073709551615",
      "data": {
        "type": "Buffer",
        "data": [
          6,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          254,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0..
        ]
      },
      "writeVersion": "1420851011205",
      "txnSignature": {
        "type": "Buffer",
        "data": [
          98,
          111,
          127,
          61,
          204,
          178,
          57,
          89,
          191,
          6,
          80,
          208,
          11,
          5,
          56,
          145,
          243,
          41,
          76,
          253,
          154,
          116,
          203,
          15,
          189,
          83,
          136,
          227,
          86,
          200,
          119,
          38,
          210,
          69,
          229,
          210,
          238,
          115,
          14...
        ]
      }
    },
    "slot": "288313982",
    "isStartup": false
  }
}

Compared to traditional RPC responses, gRPC transaction streams offer similar data with a few additional fields. A few relevant additional fields include, isVote to identify vote transactions, message headers detailing signature and signer info. The response also denotes if it is a versioned transaction or not, and it also has provisions for block, ping, and entry.

To convert this transaction received from gRPC to a traditional RPC like transactions, we use the following utility function created by Shyft.

import { TransactionFormatter } from "./utils/transaction-formatter";

const TXN_FORMATTER = new TransactionFormatter();
const txn = TXN_FORMATTER.formTransactionFromJson(
    data.transaction,
    Date.now(),
);

This function mirrors traditional Solana’s getTransaction in terms of response by returning slot, version, blocktime, meta, and transactions, making it easy to integrate with existing Solana applications.

The Following Steps — Spotting New Raydium Pools

The next step involves identifying transactions that create new pools on Raydium. Whenever a new pool is created in Raydium, a specific transaction containing the memcmp instruction is executed on the blockchain. Since we’re already streaming Raydium-related transactions in real-time, we have access to all transactions related to Raydium protocol.

However, we need to analyze these raw transactions received from the gRPC stream to identify the ones containing the memcmp instructions, and it is only possible after parsing the transactions. We have used Shyft’s transaction parsers to parse the raw Raydium transaction received from gRPC.

Press enter or click to view image in full size

Parsed Raydium Updates

After we break down the transaction (parsing), we’re left with a list of “filters”, which are essentially operations which are performed if the transaction is executed. Each filter has details like amount used in swaps(set at zero values) , the program it’s connected to (like Raydium), and the accounts involved(quoteMint, baseMint). Our job is to sift utilize accountDataSlice and see if there’s a specific one with the memcmp objects. If we find one of these, then that transaction is creating a new pool!

Adapting for Other DEXes — for programs similar to Raydium

The beauty of this pool detection method lies in its adaptability! It can be applied to other decentralized exchanges (DEXes) that utilize similar create pool (initialize) instructions for pool creation. Some examples include Meteora, Orca, and Jupiter.

The core steps remain largely the same, but with two key adjustments:

• Program ID Replacement: During the subscription request setup, you’ll need to swap out the Raydium program ID with the specific program ID of the target DEX (e.g., Meteora, Orca, Jupiter). This ensures you’re subscribing to transactions associated with that particular DEX.

Press enter or click to view image in full size

• Respective Transaction Parsers: Instead of relying on Shyft’s Raydium parsers, you’ll need to utilize the dedicated transaction parsers designed for the specific DEX you’re interested in. These parsers are essentially “code translators” optimized to understand the structure and instructions within that DEX’s transactions, and they are all available with Shyft.

By implementing these adjustments, you can effectively identify new pool creation events across various DEXes on the Solana blockchain. This empowers you to stay ahead of the curve and explore diverse pool opportunities within the DeFi ecosystem.

That’s pretty much everything about this article. In case you want to explore more about gRPCs, feel free to checkout how to setup gRPC streaming on Solana, or building a trading bot with Shyft. A big thank you for the readtime, and do stay tuned for more Defi dev insights!

If you want the entire working example, here is a git repository we have created for you to try out. Make sure you add your region-specific gRPC url, and access token to the “secrets” section before trying out.

Resources

• SHYFT API Documentation
• Shyft Website
• Get API Key
• GitHub
• Join our Discord
• Javascript SDK
• Try out our APIs on Swagger UI