Notifyx (Notification system)

Pasecinic Nichita

Distributed systems

Tech Stack

• Elixir
• NodeJS - expressjs
• Phoenix with Channels - WS API for Gateway and REST API for dedicated services
• Nebulex (for caching / service levels and gateway level)
• Postgres (will use Ecto as db wrapper with postrex adapter)
• React/TS (client test application)
• Prometheus (for metrics scraping)
• Grafana (for visual metrics monitoring)
• Docker (docker-compose for CI/deployment)

Naming is hard

Each service / component will be in a dedicated folder ⚙

• acai - Gateway with Phoenix Channels (port: 4000)
• durian - Auth Service (port: 5000)
• kiwi - Persistent Service (port: 6000)
• counter_2pc - Users notifications counter (with 2 phase commit support) (port: 2000)
• client - Client test application (port: 3333)
• guava - Mailing Service (port: 7000)
• julik - Service Discovery (port: 8000)
• nodex - Service for generating random stuff (avatars FN) (port:9000)
• monitoring - Monitoring tools configuration (grafana - port: 3000, prometheus: port: 9090) - Dashboards screenshots

Dev Notes 👀

Docker setup 🐳

docker compose up --build --force-recreate

# For cleaning up previous Docker images/containers/volumes (run in PowerShell)
# Don't need to run them on first setup
docker rmi -f $(docker images -aq)
docker rm -f $(docker ps -a -q)
docker volume rm $(docker volume ls -q)

Manual setup ⚙

Start Grafana & Prometheus Stack as separate Docker containers

cd monitoring\local
docker compose up --build

# 1. Start Gateway (acai)
cd acai && set PORT=4000&& iex --no-pry --sname gateway_node1 -S mix phx.server

# 2. Start Service Discovery (julik)
cd julik && set PORT=8000&& iex --no-pry --sname discovery_node1 -S mix phx.server

# 3. Start Auth Service (durian)
cd durian && set PORT=5000&& iex --no-pry --sname auth_node1 -S mix phx.server
cd durian && set PORT=5001&& iex --no-pry --sname auth_node2 -S mix phx.server

# 4. Start Persist Service (kiwi)
cd kiwi && set PORT=6000&& iex --no-pry --sname persist_node1 -S mix phx.server

# 5. Start Mail Service Cluster (guava)
cd guava && set ENABLE_REST_API=1& set PORT=7000&& iex --no-pry --sname mail_node1 -S mix phx.server
cd guava && set PORT=7001&& iex --no-pry --sname mail_node2 -S mix phx.server
cd guava && set PORT=7002&& iex --no-pry --sname mail_node3 -S mix phx.server
cd guava && set PORT=7003&& iex --no-pry --sname mail_node4 -S mix phx.server

# 6. Start Generator Service Cluster (nodex)
cd nodex && npm run dev:pm2
# to stop: npm run del:pm2

# 7. Start Counter 2 Phase commit service (counter_2pc)
cd counter_2pc && set PORT=2000& iex --no-pry --sname counter_2pc -S mix phx.server

# 8. Start Client application (client)
cd client && npm run dev

What should be implemented (technically) ?

Service Features:

• SQL databases (postgres) - (Auth & Persist service)
• Status endpoint /dashboard - Generated by Phoenix Framework
• Task timeouts configurable per individual task - e.g.: inside config.exs - send_email_timeout: 1000
• Service Discovery - julik
• RPC - Mailing service Nodes communicates via :erpc
• Concurrent task limit - DynamicSupervisor for mail workers has a max_children configured ^link
• Grafana / Prometheus metrics collection & monitoring
• 2 phase commit for create notification action (kiwi & counter_2pc)
  • POST - /api/notifications with is_2pc_locked: true (prepare)
  • POST - /api/notifications/commit_2pc with request_id from prepare step (commit)
  • DELETE - /api/notifications/rollback_2pc with request_id from prepare step (rollback)

Gateway Features:

• Load Balancing - Round Robin
• Outbound WS API
• Circuit breaker ^link
• Grafana / Prometheus metrics collection & monitoring
• 2 phase commit integration for services that supports it
  • 1 phase - prepare data request -> success/error
  • 2 phase - commit/rollback request -> ack/nack

The Cache:

• Implemented in Auth Service
• Implemented in Persist Service
• Replicated cache (across all Auth service nodes - durian nodes)

Other:

• Real-time events/notifications via WS API and Phoenix Channels

Some 2 Phase commit implementation notes

Services should implement several routes for it:

• POST - /api/prepare_2pc
• POST - /api/commit_2pc
• DELETE - /api/rollback_2pc

After successful prepare request it might return an identifier for the created transaction (mutation) or ack/nack:

• kiwi - will return request_id (later used to rollback/commit transaction)
• counter_2pc - just ack (user_id from request is enough to rollback/commit transaction)

Actually the terminology of commit/rollback transaction should be better called save/discard data actions. As there is no real transaction reference that could be later used to commit/rollback it, the data is still persisted somewhere and there should exist a clean-up/save handlers for each of the atomic change

The Manager2PC

Is the generic implementation for handling first and second phase from 2 phase-commit requests. The prepare phase is domain specific, meaning that it should be clearly defined all prepare tasks handlers like:

prepare_tasks = [
  Task.async(fn ->
    Services.Persist.init_2pc(socket, notification)
  end),
  Task.async(fn ->
    Services.Counter.init_2pc(socket)
  end)
]

init_2pc function should return a tuple of:

{:ok, commit_fn, rollback_fn}
{:error, data}

commit_fn and rollback_fn are as well domain specific so those should be handled by the dedicated services separately.

The second phase is executing either commit_fn or rollback_fn handlers, based on response from prepare (first) phase. A big disadvantage of 2 phase commit approach is that there is no clear definition of what should happen when a task from second phase fails (either to commit or rollback).

Note that tasks from both phases are done asynchronously with Task.async/1 and awaited with Task.await_many/2.

What should be implemented (business-wise) ?

• User could send, receive, ack notifications/messages in real-time
• Multiple & dynamic notifications topics created by users (all other operations will validate topic creators)
• Service for persist and keep track of the broadcasted notifications, topics, subscriptions
• Service for sending notifications via email
• Service for generating PNG avatars