Using Signal to monitor experiments on the go

In her blog post, Claire Wang mentions a cool way to monitor her experiments from her phone: make a Discord server for yourself and send requests to the server’s webhooks. You could use this to send updates from your experiments, like when they have finished running or if they have failed for some reason, and receive notifications on your phone1. This is great, since I have mostly done one of the following things to monitor my experiments.

  1. Split my screen between my terminal and Netflix, then try watching both at the same time.

  2. Keep the terminal open on my laptop in fullscreen while I play my ukulele.

  3. SSH into the server from my phone2 and take a look at the logs while having dinner.

  4. Check on the experiment every 30 minutes or an hour (but only as often as I remember to).

It is easy to see that these methods are both inefficient and annoying. The problem is that I am not comfortable with using Discord3. I do, however, use Signal, so I came up with an easy way to use Signal to monitor my experiments instead. Here is how.

Set up

Install signal-cli on the machine where you want to trigger the notifications. You can download the precompiled binary and add it to your PATH if you do not have root access on the machine. You can then follow the instructions in the README to link signal-cli to your Signal account. Since I already had an account that I used from my phone, I used signal-cli link to link signal-cli to my account.

Usage

There are a lot of ways to use signal-cli to send messages. What I did was to create a new Signal group with only myself in it (aptly named “Alerts”), and send messages to that group. I needed the group ID to send messages to the group, which I got by running signal-cli listGroups. Because I created the group after linking signal-cli to my account, I had to run signal-cli sendSyncRequest to synchronize signal-cli with my account before I could see the new group in the list of groups. All I had to do now was to run the command below.

signal-cli -a <my_phone_number> send -m "<the message>" -g <group_id> --notify-self

Note that <my_phone_number> is the phone number of the account linked to signal-cli. The --notify-self flag is important, since it triggers a notification for you even though the message is sent by your own account. If you are using different accounts for triggering and receiving alerts, you may not need to use this flag. Instead of creating a separate group for alerts, you could send messages to the Note to Self chat by passing --note-to-self instead of -g <group_id>. As I said before, there are a lot of ways to do this.

Practice

I actually created small executable scripts to wrap some of the signal-cli commands. This makes it simpler for both my coding agents and me to trigger alerts. Once I have added these scripts to my PATH, I can use simple commands like sigalert "Experiment finished!" to trigger alerts from my experiment scripts. I also created a script to send messages to any other contacts by referring to their predefined aliases, like sigmsg alice "Hey, Mrigank's experiment has finished!"4. While using coding agents, I can add an instruction to send me messages about their progress in executing a long-running task. I can also automatically spawn a coding agent once an experiment has finished, and have it send me a summary of the results5.

  1. Why must you monitor your experiments? From my experience, experiments can fail for a variety of reasons, sometimes random and sometimes due to bugs in the code. If I am starting an experiment that I would expect to run for a day or two, I would like to know if it has failed as soon as possible so that I can fix the issue and restart it. It may even not be possible to predict the time an experiment will take to run, for example, when using the asynchronous batch API offered by some LLM providers. It is indeed helpful to know when and how an experiment has terminated. 

  2. There’s a nice app called Termius if you really want to do this. 

  3. Let’s just say that Discord’s data collection practices, user data handling, and prior security incidents are not the most reassuring. 

  4. Alright, I admit this is not how I intended to use this. 

  5. I got a bit carried away and vibe-coded a system to connect signal-cli with kiro-cli for automatically responding to incoming messages (basically a chatbot). Special shout-out to Kiro—my favorite coding agent nowadays for its nice model selection at a very reasonable price.