Chapter 10 Fine-tune trading strategy per symbol

10.1 Objectives

  • describe and design the required functionality
  • add docker to project
  • set up ecto inside the naive app
  • create and migrate the DB
  • seed symbols’ settings
  • update the Naive.Leader to fetch settings

10.2 Describe and design the required functionality

At this moment the settings of our naive strategy are hardcoded inside the Naive.Leader:

  # /apps/naive/lib/naive/leader.ex
  ...
  defp fetch_symbol_settings(symbol) do
    symbol_filters = fetch_symbol_filters(symbol)

    Map.merge(
      %{
        symbol: symbol,              # <=
        chunks: 5,                   # <=
        budget: 100,                 # <=
        # -0.01% for quick testing   # <=
        buy_down_interval: "0.0001", # <= all of those settings
        # -0.12% for quick testing   # <=
        profit_interval: "-0.0012",  # <=
        rebuy_interval: "0.001"      # <=
      },
      symbol_filters
    )
  end
  ...

The problem about those is that they are hardcoded and there’s no flexibility to define them per symbol at the moment.

In this chapter, we will move them out from this file into the Postgres database.

10.3 Add docker to project

The requirements for this section are docker and docker-compose installed in your system.

Inside the main directory of our project create a new file called docker-compose.yml and fill it with the below details:

# /docker-compose.yml
version: "3.2"
services:
  db:
    image: postgres:latest
    restart: always
    environment:
      POSTGRES_PASSWORD: "hedgehogSecretPassword"
    ports:
      - 5432:5432
    volumes:
      - ../postgres-data:/var/lib/postgresql/data

If you are new to docker here’s the gist of what the above will do:

  • it will start a single service called “db”
  • “db” service will use the latest version of the postgres (image) inside the docker container (latest version as tagged per https://hub.docker.com/_/postgres/)
  • we map TCP port 5432 in the container to port 5432 on the Docker host(format container_port:hosts_port)
  • we set up environmental variable inside the docker container that will be used by the Postgres app as a password for the default (postgres) user
  • volumes option maps the directory from inside of the container to the host. This way we will keep the state of the database between restarts.

We can now start the service using docker-compose:

$ docker-compose up -d
Creating hedgehog_db_1 ... done

To validate that it works we can run:

$ docker ps -a
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS
PORTS                    NAMES
98558827b80b   postgres:latest   "docker-entrypoint.sh"   4 seconds ago   Up 4 seconds
0.0.0.0:5432->5432/tcp   hedgehog_db_1

10.4 Set up ecto inside the naive app

Let’s start by adding database access to the naive application. The first step is to add the Ecto module together with the Postgrex ecto’s driver package to the deps function inside the mix.exs file. As we are going to use Enums inside Postgres, we need to add the EctoEnum module as well:

  # /apps/naive/mix.exs
  defp deps do
    [
      {:binance, "~> 1.0"},
      {:binance_mock, in_umbrella: true},
      {:decimal, "~> 2.0"},
      {:ecto_sql, "~> 3.0"},     # <= New line
      {:ecto_enum, "~> 1.4"},    # <= New line
      {:phoenix_pubsub, "~> 2.0"},
      {:postgrex, ">= 0.0.0"},   # <= New line
      {:streamer, in_umbrella: true}
    ]
  end

Remember about installing those deps using:

$ mix deps.get

We can now use the ecto generator to add an the ecto repository to the Naive application:

$ cd apps/naive
$ mix ecto.gen.repo -r Naive.Repo
* creating lib/naive
* creating lib/naive/repo.ex
* updating ../../config/config.exs
Don't forget to add your new repo to your supervision tree
(typically in lib/naive/application.ex):

    {Naive.Repo, []}

And to add it to the list of Ecto repositories in your
configuration files (so Ecto tasks work as expected):

    config :naive,
      ecto_repos: [Naive.Repo]

Back to the IDE, the generator updated our config/config.exs file with the default access details to the database, we need to modify them to point to our Postgres docker instance as well as add a list of ecto repositories for our naive app (as per instruction above):

# /config/config.exs
config :naive,                # <= added line
  ecto_repos: [Naive.Repo],   # <= added line
  binance_client: BinanceMock # <= merged from existing config

config :naive, Naive.Repo,
  database: "naive",                  # <= updated
  username: "postgres",               # <= updated
  password: "hedgehogSecretPassword", # <= updated
  hostname: "localhost"

Here we can use localhost as inside the docker-compose.yml file we defined port forwarding from the container to the host(Postgres is available at localhost:5432). We also merged the existing binance_client setting together with the new ecto_repos setting.

The last step to be able to communicate with the database using Ecto will be to add the Naive.Repo module(created by generator) to the children list of the Naive.Application:

# /apps/naive/lib/naive/application.ex
...
  def start(_type, _args) do
    children = [
      {Naive.Repo, []}, # <= added line
      {
        DynamicSupervisor,
        strategy: :one_for_one, name: Naive.DynamicSymbolSupervisor
      }
    ]
    ...

10.5 Create and migrate the DB

We can now create a new naive database using the mix tool, after that we will be able to generate a migration file that will create the settings table:

$ mix ecto.create -r Naive.Repo
The database for Naive.Repo has been created
$ cd apps/naive
$ mix ecto.gen.migration create_settings
* creating priv/repo/migrations
* creating priv/repo/migrations/20210202223209_create_settings.exs

We can now copy the current hardcoded settings from the Naive.Leader module and use them as a column list of our new settings table. All of the below alterations need to be done inside the change function of our migration file:

# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
  def change do
    create table(:settings) do
      add(:symbol, :text, null: false)
      add(:chunks, :integer, null: false)
      add(:budget, :decimal, null: false)
      add(:buy_down_interval, :decimal, null: false)
      add(:profit_interval, :decimal, null: false)
      add(:rebuy_interval, :decimal, null: false)
    end
  end

At this moment we just copied the settings and converted them to columns using the add function. We need now to take care of the id column. We need to pass primary_key: false to the create table macro to stop it from creating the default integer-based id column. Instead of that we will define the id column ourselves with :uuid type and pass a flag that will indicate that it’s the primary key of the settings table:

# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(:settings, primary_key: false) do
      add(:id, :uuid, primary_key: true)
      ...

We will also add create and update timestamps that come as a bundle when using the timestamps() function inside the create table macro:

# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(...) do
      ...

      timestamps() # <= both create and update timestamps
    end
    ...

We will add a unique index on the symbol column to avoid any possible duplicates:

# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(...) do
      ...
    end

    create(unique_index(:settings, [:symbol]))
  end
...

We will now add the status field which will be an Enum. It will be defined inside a separate file and alias’ed from our migration, this way we will be able to use it from within the migration and the inside the lib code. First, we will apply changes to our migration and then we will move on to creating the Enum module. Here’s the full implementation of migration for reference:

# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
defmodule Naive.Repo.Migrations.CreateSettings do
  use Ecto.Migration

  alias Naive.Schema.TradingStatusEnum

  def change do
    TradingStatusEnum.create_type()

    create table(:settings, primary_key: false) do
      add(:id, :uuid, primary_key: true)
      add(:symbol, :text, null: false)
      add(:chunks, :integer, null: false)
      add(:budget, :decimal, null: false)
      add(:buy_down_interval, :decimal, null: false)
      add(:profit_interval, :decimal, null: false)
      add(:rebuy_interval, :decimal, null: false)
      add(:status, TradingStatusEnum.type(), default: "off", null: false)

      timestamps()
    end

    create(unique_index(:settings, [:symbol]))
  end
end

That finishes our work on the migration file. We will now focus on TradingStatusEnum implementation. First, we need to create a schema directory inside the apps/naive/lib/naive directory and file called trading_status_enum.ex and place below logic (defining the enum) in it:

# /apps/naive/lib/naive/schema/trading_status_enum.ex
import EctoEnum

defenum(Naive.Schema.TradingStatusEnum, :trading_status, [:on, :off])

We used the defenum macro from the ecto_enum module to define our enum. It’s interesting to point out that we didn’t need to define a new module as defenum macro takes care of that for us.

Let’s run the migration to create the table, unique index, and the enum:

$ mix ecto.migrate
00:51:16.757 [info]  == Running 20210202223209 Naive.Repo.Migrations.CreateSettings.change/0
forward
00:51:16.759 [info]  execute "CREATE TYPE public.trading_status AS ENUM ('on', 'off')"
00:51:16.760 [info]  create table settings
00:51:16.820 [info]  create index settings_symbol_index
00:51:16.829 [info]  == Migrated 20210202223209 in 0.0s

We can now create a schema file for the settings table so inside the /apps/naive/lib/naive/schema create a file called settings.ex. We will start with a skeleton implementation of schema file together with the copied list of columns from the migration and convert to ecto’s types using it’s docs:

# /apps/naive/lib/naive/schema/settings.ex
defmodule Naive.Schema.Settings do
  use Ecto.Schema

  alias Naive.Schema.TradingStatusEnum

  @primary_key {:id, :binary_id, autogenerate: true}

  schema "settings" do
    field(:symbol, :string)
    field(:chunks, :integer)
    field(:budget, :decimal)
    field(:buy_down_interval, :decimal)
    field(:profit_interval, :decimal)
    field(:rebuy_interval, :decimal)
    field(:status, TradingStatusEnum)

    timestamps()
  end
end

10.6 Seed symbols’ settings

So we have all the pieces of implementation to be able to create DB, migrate the settings table, and query it using Ecto. To be able to drop the hardcoded settings from the Naive.Leader we will need to fill our database with the “default” setting for each symbol. To achieve that we will define default settings inside the config/config.exs file and we will create a seed script that will fetch all pairs from Binance and insert a new config row inside DB for each one.

Let’s start by adding those default values to the config file(we will merge them into the structure defining binance_client and ecto_repos):

# config/config.exs
config :naive,
  ecto_repos: [Naive.Repo],
  binance_client: BinanceMock,
  trading: %{
    defaults: %{
      chunks: 5,
      budget: 1000,
      buy_down_interval: "0.0001",
      profit_interval: "-0.0012",
      rebuy_interval: "0.001"
    }
  }

Moving on to the seeding script, we need to create a new file called seed_settings.exs inside the
/apps/naive/lib/naive/priv/ directory. Let’s start by aliasing the required modules and requiring the Logger:

# /apps/naive/priv/seed_settings.exs
require Logger

alias Naive.Repo
alias Naive.Schema.Settings

Next, we will get the Binance client from the config:

# /apps/naive/priv/seed_settings.exs
...
binance_client = Application.compile_env(:naive, :binance_client)

Now, it’s time to fetch all the symbols(pairs) that Binance supports:

# /apps/naive/priv/seed_settings.exs
...
Logger.info("Fetching exchange info from Binance to create trading settings")

{:ok, %{symbols: symbols}} = binance_client.get_exchange_info()

Now we need to fetch default trading settings from the config file as well as the current timestamp:

# /apps/naive/priv/seed_settings.exs
...
%{
  chunks: chunks,
  budget: budget,
  buy_down_interval: buy_down_interval,
  profit_interval: profit_interval,
  rebuy_interval: rebuy_interval
} = Application.compile_env(:naive, :trading).defaults

timestamp = NaiveDateTime.utc_now()
  |> NaiveDateTime.truncate(:second)

We will use the default settings for all rows to be able to insert data into the database. Normally we wouldn’t need to set inserted_at and updated_at fields as Ecto would generate those values for us when using Repo.insert/2 but we won’t be able to use this functionality as it takes a single record at the time. We will be using Repo.insert_all/3 which is a bit more low-level function without those nice features like filling timestamps(sadly). Just to be crystal clear - Repo.insert/2 takes at least a couple of seconds(on my machine) for 1000+ symbols currently supported by Binance, on the other hand Repo.insert_all/3, will insert all of them in a couple of hundred milliseconds.

As our structs will differ by only the symbol column we can first create a full struct that will serve as a template:

# /apps/naive/priv/seed_settings.exs
...
base_settings = %{
  symbol: "",
  chunks: chunks,
  budget: Decimal.new(budget),
  buy_down_interval: Decimal.new(buy_down_interval),
  profit_interval: Decimal.new(profit_interval),
  rebuy_interval: Decimal.new(rebuy_interval),
  status: "off",
  inserted_at: timestamp,
  updated_at: timestamp
}

We will now map each of the retrieved symbols and inject them to the base_settings structs and pushing all of those to the Repo.insert_all/3 function:

# /apps/naive/priv/seed_settings.exs
...
Logger.info("Inserting default settings for symbols")

maps = symbols
  |> Enum.map(&(%{base_settings | symbol: &1["symbol"]}))

{count, nil} = Repo.insert_all(Settings, maps)

Logger.info("Inserted settings for #{count} symbols")

10.7 Update the Naive.Leader to fetch settings

The final step will be to update the Naive.Leader to fetch the settings from the database. At the top of the module add the following:

# /apps/naive/lib/naive/leader.ex
  ...
  alias Naive.Repo
  alias Naive.Schema.Settings
  ...

Now we need to modify the fetch_symbol_settings/1 to fetch settings from DB instead of the hardcoded map. We will use Repo.get_by!/3 as we are unable to trade without settings. The second trick used here is Map.from_struct/1 that is required here as otherwise result would become the Naive.Schema.Settings struct(this would cause problems further down the line as we are iterating on the returned map and would get the protocol Enumerable not implemented for %Naive.Schema.Settings error):

# /apps/naive/lib/naive/leader.ex
  ...
  defp fetch_symbol_settings(symbol) do
    symbol_filters = fetch_symbol_filters(symbol)
    settings = Repo.get_by!(Settings, symbol: symbol)

    Map.merge(
      symbol_filters,
      settings |> Map.from_struct()
    )
  end
  ...

We can now run the seeding script to fill our database with the default settings:

$ cd apps/naive
$ mix run priv/seed_settings.exs
18:52:29.341 [info]  Fetching exchange info from Binance to create trading settings
18:52:31.571 [info]  Inserting default settings for symbols
18:52:31.645 [info]  Inserted settings for 1276 symbols

We can verify that records were indeed inserted into the database by connecting to it using the psql application:

$ psql -Upostgres -hlocalhost
Password for user postgres: # <= use 'postgres' password here
...
postgres=# \c naive
You are now connected to database "naive" as user "postgres".
naive=# \x
Expanded display is on.
naive=# SELECT * FROM settings;
-[ RECORD 1 ]-----+-------------------------------------
id                | 159c8f32-d571-47b2-b9d7-38bb42868043
symbol            | ETHUSDT
chunks            | 5
budget            | 1000
buy_down_interval | 0.0001
profit_interval   | -0.0012
rebuy_interval    | 0.001
status            | off
inserted_at       | 2021-02-02 18:52:31
updated_at        | 2021-02-02 18:52:31

# press arrows to scroll, otherwise press `q`

naive=# SELECT COUNT(*) FROM settings;
-[ RECORD 1 ]
count | 1276

naive=# \q # <= to close the `psql`

That confirms that there are 1276 settings inside the database that will allow us to continue trading which we can check by running our app inside the IEx(from the main project’s directory):

$ iex -S mix
...
iex(1)> Naive.start_trading("NEOUSDT")
19:20:02.936 [info]  Starting new supervision tree to trade on NEOUSDT
{:ok, #PID<0.378.0>}
19:20:04.584 [info]  Initializing new trader(1612293637000) for NEOUSDT

The above log messages confirm that the Naive.Leader was able to fetch settings from the database that were later put into the Naive.Trader’s state and passed to it.

[Note] Please remember to run the mix format to keep things nice and tidy.

The source code for this chapter can be found on GitHub