Chapter 9 Fine-tune trading strategy per symbol
In the previous chapter, we achieved a major milestone: our trading strategy can now dynamically scale from 1 to 5 parallel traders as the price moves. But all those traders share the same hardcoded settings - the same budget, the same intervals, the same profit targets.
That’s a problem. Different trading pairs have wildly different characteristics. A volatile meme coin might need wider intervals to avoid getting wiped, while a stable blue-chip pair could use tighter settings. And right now, if we want to change any parameter, we have to edit code and redeploy.
In this chapter, we’ll fix this by moving our settings out of the code and into a Postgres database. We’ll set up Docker to run Postgres, add Ecto to our project, create a settings table, and seed it with default values for every trading pair Binance supports. By the end, each symbol will have its own configurable settings that we can tweak without touching code.
Let’s make our strategy configurable!
9.1 Objectives
- describe and design the required functionality
- add docker to project
- set up Ecto inside the
naiveapp - create and migrate the DB
- seed symbols’ settings
- update the
Naive.Leaderto fetch settings
9.2 Describe and design the required functionality
At this moment, the settings of our naive strategy are hardcoded inside the Naive.Leader:
# /apps/naive/lib/naive/leader.ex
...
defp fetch_symbol_settings(symbol) do
symbol_filters = fetch_symbol_filters(symbol)
Map.merge(
%{
symbol: symbol, # <=
chunks: 5, # <=
budget: 100, # <=
# -0.01% for quick testing # <=
buy_down_interval: "0.0001", # <= all of those settings
# -0.12% for quick testing # <=
profit_target: "-0.0012", # <=
rebuy_interval: "0.001" # <=
},
symbol_filters
)
end
...The problem is that they’re hardcoded - there’s no flexibility to define different values per symbol.
In this chapter, we will move them out from this file into the Postgres database.
9.3 Add docker to project
The requirement for this section is docker installed on your system.
Inside the main directory of our project create a new file called docker-compose.yml and fill it with the below details:
# /docker-compose.yml
services:
db:
image: postgres:latest
restart: always
environment:
POSTGRES_PASSWORD: "hedgehogSecretPassword"
ports:
- 5432:5432
volumes:
- ../postgres-data:/var/lib/postgresql/dataIf you are new to Docker, here’s the gist of what the above will do:
- it will start a single service called “db”
- “db” service will use the
latestversion of thepostgres(image) inside the docker container (latestversion as tagged per https://hub.docker.com/_/postgres/) - we map TCP port 5432 in the container to port 5432 on the Docker host(format: host_port:container_port)
- we set up an environment variable inside the Docker container that will be used by Postgres as the password for the default (
postgres) user volumesoption maps the directory from inside of the container to the host. This way we will keep the state of the database between restarts.
We can now start the service using docker compose:
To validate that it works we can run:
9.4 Set up Ecto inside the naive app
Let’s start by adding database access to the naive application. The first step is to add the Ecto module together with the Postgrex ecto’s driver package to the deps function inside the mix.exs file. As we are going to use Enums inside Postgres, we need to add the EctoEnum module as well:
# /apps/naive/mix.exs
defp deps do
[
{:binance_mock, in_umbrella: true},
{:binance, "~> 1.0"},
{:decimal, "~> 2.0"},
{:ecto_sql, "~> 3.0"}, # <= New line
{:ecto_enum, "~> 1.4"}, # <= New line
{:phoenix_pubsub, "~> 2.0"},
{:postgrex, ">= 0.0.0"}, # <= New line
{:streamer, in_umbrella: true}
]
endRemember about installing those deps using:
We can now use the Ecto generator to add a repository to the Naive application:
$ cd apps/naive
$ mix ecto.gen.repo -r Naive.Repo
* creating lib/naive
* creating lib/naive/repo.ex
* updating ../../config/config.exs
Don't forget to add your new repo to your supervision tree
(typically in lib/naive/application.ex):
{Naive.Repo, []}
And to add it to the list of Ecto repositories in your
configuration files (so Ecto tasks work as expected):
config :naive,
ecto_repos: [Naive.Repo]Back to the IDE, the generator updated our config/config.exs file with the default access details to the database, we need to modify them to point to our Postgres docker instance as well as add a list of ecto repositories for our naive app (as per instruction above):
# /config/config.exs
config :naive, Naive.Repo,
database: "naive", # <= updated
username: "postgres", # <= updated
password: "postgres", # <= updated
hostname: "localhost"
...
config :naive,
ecto_repos: [Naive.Repo], # <= added line
binance_client: BinanceMockHere we can use localhost as inside the docker-compose.yml file we defined port forwarding from the container to the host(Postgres is available at localhost:5432). We also merged the existing binance_client setting together with the new ecto_repos setting.
The last step to be able to communicate with the database using Ecto will be to add the Naive.Repo module(created by generator) to the children list of the Naive.Application:
9.5 Create and migrate the DB
We can now create a new naive database using the mix tool, after that we will be able to generate a migration file that will create the settings table:
$ mix ecto.create -r Naive.Repo
The database for Naive.Repo has been created
$ cd apps/naive
$ mix ecto.gen.migration create_settings
* creating priv/repo/migrations
* creating priv/repo/migrations/20210202223209_create_settings.exsWe can now copy the current hardcoded settings from the Naive.Leader module and use them as a column list of our new settings table. All of the below alterations need to be done inside the change function of our migration file:
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
def change do
create table(:settings) do
add(:symbol, :text, null: false)
add(:chunks, :integer, null: false)
add(:budget, :decimal, null: false)
add(:buy_down_interval, :decimal, null: false)
add(:profit_target, :decimal, null: false)
add(:rebuy_interval, :decimal, null: false)
end
endAt this moment we just copied the settings and converted them to columns using the add function. We need now to take care of the id column. We need to pass primary_key: false to the create table macro to stop it from creating the default integer-based id column. Instead of that we will define the id column ourselves with :uuid type and pass a flag that will indicate that it’s the primary key of the settings table:
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
create table(:settings, primary_key: false) do
add(:id, :uuid, primary_key: true)
...We will also add create and update timestamps that come as a bundle when using the timestamps() function inside the create table macro:
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
create table(...) do
...
timestamps() # <= both create and update timestamps
end
...We will add a unique index on the symbol column to avoid any possible duplicates:
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
create table(...) do
...
end
create(unique_index(:settings, [:symbol]))
end
...We will now add the status field which will be an Enum. It will be defined inside a separate file and alias’ed from our migration, this way we will be able to use it from within the migration and the inside the lib code. First, we will apply changes to our migration and then we will move on to creating the Enum module.
Here’s the full implementation of migration for reference:
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
defmodule Naive.Repo.Migrations.CreateSettings do
use Ecto.Migration
alias Naive.Schema.TradingStatusEnum
def change do
TradingStatusEnum.create_type()
create table(:settings, primary_key: false) do
add(:id, :uuid, primary_key: true)
add(:symbol, :text, null: false)
add(:chunks, :integer, null: false)
add(:budget, :decimal, null: false)
add(:buy_down_interval, :decimal, null: false)
add(:profit_target, :decimal, null: false)
add(:rebuy_interval, :decimal, null: false)
add(:status, TradingStatusEnum.type(), default: "off", null: false)
timestamps()
end
create(unique_index(:settings, [:symbol]))
end
endThat finishes our work on the migration file. We will now focus on TradingStatusEnum implementation. First, we need to create a schema directory inside the apps/naive/lib/naive directory and file called trading_status_enum.ex and place below logic (defining the enum) in it:
# /apps/naive/lib/naive/schema/trading_status_enum.ex
import EctoEnum
defenum(Naive.Schema.TradingStatusEnum, :trading_status, [:on, :off])We used the defenum macro from the ecto_enum module to define our enum. It’s interesting to point out that we didn’t need to define a new module as defenum macro takes care of that for us.
Let’s run the migration to create the table, unique index, and the enum:
$ mix ecto.migrate
00:51:16.757 [info] == Running 20210202223209 Naive.Repo.Migrations.CreateSettings.change/0
forward
00:51:16.759 [info] execute "CREATE TYPE public.trading_status AS ENUM ('on', 'off')"
00:51:16.760 [info] create table settings
00:51:16.820 [info] create index settings_symbol_index
00:51:16.829 [info] == Migrated 20210202223209 in 0.0sWe can now create a schema file for the settings table so inside the /apps/naive/lib/naive/schema create a file called settings.ex. We will start with a skeleton implementation of schema file together with the copied list of columns from the migration and convert to ecto’s types using it’s docs:
# /apps/naive/lib/naive/schema/settings.ex
defmodule Naive.Schema.Settings do
use Ecto.Schema
alias Naive.Schema.TradingStatusEnum
@primary_key {:id, :binary_id, autogenerate: true}
schema "settings" do
field(:symbol, :string)
field(:chunks, :integer)
field(:budget, :decimal)
field(:buy_down_interval, :decimal)
field(:profit_target, :decimal)
field(:rebuy_interval, :decimal)
field(:status, TradingStatusEnum)
timestamps()
end
end9.6 Seed symbols’ settings
So we have all the pieces of implementation to be able to create DB, migrate the settings table, and query it using Ecto. To be able to drop the hardcoded settings from the Naive.Leader we will need to fill our database with the “default” setting for each symbol. To achieve that we will define default settings inside the config/config.exs file and we will create a seed script that will fetch all pairs from Binance and insert a new config row inside DB for each one.
Let’s start by adding those default values to the config file(we will merge them into the structure defining binance_client and ecto_repos):
# config/config.exs
config :naive,
ecto_repos: [Naive.Repo],
binance_client: BinanceMock,
trading: %{
defaults: %{
chunks: 5,
budget: 1000,
buy_down_interval: "0.0001",
profit_target: "-0.0012",
rebuy_interval: "0.001"
}
}Moving on to the seeding script, we need to create a new file called seed_settings.exs inside the /apps/naive/priv/ directory.
Let’s start by aliasing the required modules and requiring the Logger:
Next, we will get the Binance client from the config:
# /apps/naive/priv/seed_settings.exs
...
binance_client = Application.compile_env(:naive, :binance_client)Now, it’s time to fetch all the symbols(pairs) that Binance supports:
# /apps/naive/priv/seed_settings.exs
...
Logger.info("Fetching exchange info from Binance to create trading settings")
{:ok, %{symbols: symbols}} = binance_client.get_exchange_info()Now we need to fetch default trading settings from the config file as well as the current timestamp:
# /apps/naive/priv/seed_settings.exs
...
%{
chunks: chunks,
budget: budget,
buy_down_interval: buy_down_interval,
profit_target: profit_target,
rebuy_interval: rebuy_interval
} = Application.compile_env(:naive, :trading).defaults
timestamp = NaiveDateTime.utc_now()
|> NaiveDateTime.truncate(:second)We will use the default settings for all rows to be able to insert data into the database.
Normally we wouldn’t need to set inserted_at and updated_at fields as Ecto would generate those values for us
when using Repo.insert/2 but we won’t be able to use this functionality as it takes a single record at a time.
We will be using Repo.insert_all/3 which is a bit more low-level function without those nice features like filling timestamps(sadly).
Just to be crystal clear - Repo.insert/2 takes at least a couple of seconds(on my machine) for thousands of symbols currently supported by Binance,
on the other hand Repo.insert_all/3, will insert all of them in a couple of hundred milliseconds.
As our structs will differ by only the symbol column we can first create a full struct that will serve as a template:
# /apps/naive/priv/seed_settings.exs
...
base_settings = %{
symbol: "",
chunks: chunks,
budget: Decimal.new(budget),
buy_down_interval: Decimal.new(buy_down_interval),
profit_target: Decimal.new(profit_target),
rebuy_interval: Decimal.new(rebuy_interval),
status: "off",
inserted_at: timestamp,
updated_at: timestamp
}We will now prepare and insert 1000 rows at a time.
Each row will be the base_settings struct with injected symbol.
Using Enum.reduce/3 will enable us to sum each result of the Repo.insert_all/3 function call:
# /apps/naive/priv/seed_settings.exs
...
Logger.info("Inserting default settings for symbols")
total_count = symbols
|> Enum.map(&(%{base_settings | symbol: &1["symbol"]}))
|> Enum.chunk_every(1000)
|> Enum.reduce(0, fn batch, acc ->
{count, nil} = Repo.insert_all(Settings, batch)
Logger.info("Inserted batch of #{count} symbols")
acc + count
end)
Logger.info("Inserted settings for #{total_count} symbols")9.7 Update the Naive.Leader to fetch settings
The final step will be to update the Naive.Leader to fetch the settings from the database. At the top of the module add the following:
Now we need to modify the fetch_symbol_settings/1 to fetch settings from DB instead of the hardcoded map.
We will use Repo.get_by!/2 as we are unable to trade without settings.
The second trick used here is Map.from_struct/1 that is required here as otherwise
result would become the Naive.Schema.Settings struct
(this would cause problems further down the line as we are iterating on the returned map and
would get the protocol Enumerable not implemented for %Naive.Schema.Settings error):
9.8 Manual testing in IEx and psql
We can now run the seeding script to fill our database with the default settings:
$ cd apps/naive
$ mix run priv/seed_settings.exs
18:52:29.341 [info] Fetching exchange info from Binance to create trading settings
18:52:31.571 [info] Inserting default settings for symbols
18:52:31.614 [info] Inserted batch of 1000 symbols
18:52:31.642 [info] Inserted batch of 276 symbols
18:52:31.645 [info] Inserted settings for 1276 symbolsWe can verify that records were indeed inserted into the database by connecting to it using the psql application:
$ psql -Upostgres -hlocalhost
Password for user postgres: # <= use 'postgres' password here
...
postgres=# \c naive
You are now connected to database "naive" as user "postgres".
naive=# \x
Expanded display is on.
naive=# SELECT * FROM settings;
-[ RECORD 1 ]-----+-------------------------------------
id | 159c8f32-d571-47b2-b9d7-38bb42868043
symbol | XRPUSDT
chunks | 5
budget | 1000
buy_down_interval | 0.0001
profit_target | -0.0012
rebuy_interval | 0.001
status | off
inserted_at | 2021-02-02 18:52:31
updated_at | 2021-02-02 18:52:31
# press arrows to scroll, otherwise press `q`
naive=# SELECT COUNT(*) FROM settings;
-[ RECORD 1 ]
count | 1276
naive=# \q # <= to close the `psql`That confirms that there are 1276 settings inside the database that will allow us to continue trading which we can check by running our app inside the IEx(from the main project’s directory):
$ iex -S mix
...
iex(1)> Streamer.start_streaming("XRPUSDT")
{:ok, #PID<0.313.0>}
iex(2)> Naive.start_trading("XRPUSDT")
19:20:02.936 [info] Starting new supervision tree to trade on XRPUSDT
{:ok, #PID<0.378.0>}
19:20:04.584 [info] Initializing new trader(1612293637000) for XRPUSDTThe above log messages confirm that the Naive.Leader was able to fetch settings from the database that were later put into the Naive.Trader’s state and passed to it.
Settings are now database-driven! This opens up a lot of possibilities:
- Each symbol can have its own unique trading parameters
- We can adjust settings without redeploying code
- We’ve added a
statusfield that will let us enable/disable trading per symbol - The seeding script makes it easy to populate settings for all Binance pairs
So, What’s Next?
We’ve made the naive trading app configurable, but the streamer app is still completely manual.
Every time we restart the application, we have to call Streamer.start_streaming/1 for each symbol we want to trade.
In the next chapter, we’ll give the streamer the same treatment: add supervision, database-backed settings, and autostart functionality. When we start our app, it will automatically resume streaming on all previously enabled symbols. We’ll also add the ability to stop streaming - something we can’t even do right now!
[Note] Please remember to run mix format to keep things nice and tidy.
The source code for this chapter can be found in the book’s source code repository (branch: chapter_09).