Joel, (always one syllable), often "@jayroh"


Weeknotes for the week ending May.13

13 May 2022

In keeping with the theme of sharing misery in the hopes that maybe someone, somewhere, runs into this problem on Google some day and can save themselves from hours of struggling with “what the hell is happening here, exactly?”

… I present you a meditation on how Unifi’s IPS (intrusion prevention system) can ruin your day.

I was home. I was toying around with some stuff. After spinning up some hosts on a cloud provider I tried to ssh-copy-id a key into said hosts. Those worked just fine. Subsequent attempts to ssh into the machine did not. It just sat there. Hanging. Frozen. Doin’ nothing. Why?

I spent a little while troubleshooting. Thinking maybe the hosts had ufw turned on? Did an Ansible script I ran have something in it that I didn’t consider? Etc. All answers were “no”. Huh.

What does nmap say about this?

$ nmap -Pn -p22 host-address
Starting Nmap 7.92 ( ) at 2022-05-09 10:21 EDT
Nmap scan report for host-address (ip)
Host is up.

22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 2.08 seconds

filtered? Why?

What about another machine on my network?

$ nmap -Pn -p22 host-address
Starting Nmap 7.92 ( ) at 2022-05-09 10:21 EDT
Nmap scan report for host-address (ip)
Host is up.

22/tcp open     ssh

Nmap done: 1 IP address (1 host up) scanned in 2.08 seconds

open? What the…? Laptop on my home network - “no-go”. Little homelab server - “a-ok”. So it’s something on my network?

Yes. I’ll cut to the chase.

If you run a Unifi rig at home, and you enable Unifi’s “IPS”, then you might have inadvertently enabled some security rules that prevent doing something like ssh-copy-id. Your “event timeline” might look something like the following:

Screenshot of what looked wrong

You can click the button on the right to allow from your/all ip’s, or you can turn off this particular rule via:

Settings ▶
  Firewall & Security ▶
    System Sensitivity ▶
      Edit threat categories ▶
        ☑️ Scan

I should note, this might be only for the UDM Pro. Kyle let me know he doesn’t see this in his UDM.

Happy networking!

Weeknotes for the week ending Apr.22

22 Apr 2022

Revoke LetsEncrypt Certs in Traefik’s acme.json File

First, credit where credit is due to Daniel and Senan in this Traefik issue. To them I say “thank you”.

Second, a little context. In the past few months I’ve been tinkering with self-hosting some of my own apps, as well as some services at home. Among the things I’ve been experimenting with are Traefik, a “cloud native proxy”. One of the nice things Traefik does is generates and sets up LetsEncrypt certificates if you set all the things up correctly. It takes some tinkering but once you get it, it’s really nice.

Now, all of the above is true until you potentially run into the time where you need to revoke one of those certs. Perhaps you would like to move to another host? Or switch to Nginx? Searching for a solution to that will result in many pages talking about the recent forced-revocation of certificates:

Let’s Encrypt sent me an email that our certificates which were checked with a CAA record will be revoked tomorrow because of a bug in the implementation.

… lots of that.

So - how do you revoke and delete the cert Traefik had generated?

The answer lies in extracting the key and cert from the generated acme.json file generated by Traefik. To do so you can utilize

  1. Utilize the jq tool to dig into that JSON file
  2. Pipe into base64 to decode it and
  3. And then redirect STDOUT to a tmp file.

In the following example, my LetsEncrypt info is nested within the dnsimple key as they are the DNS provider I use to resolve LetsEncrypt requests. Also is neither my domain, nor one that’s registered (Yet? So have at it).

cat acme.json \
  | jq -r ".dnsimple .Certificates | .[] | select(.domain.main == \"\").key" \
  | base64 --decode \
  > /tmp/tmp.key

cat acme.json \
  | jq -r ".dnsimple .Certificates | .[] | select(.domain.main == \"\").certificate" \
  | base64 --decode \
  > /tmp/tmp.cert

After generating those two temp files you can revoke, and optionally delete, your domain’s cert with the following:

letsencrypt revoke \
  -d "" \
  --key-path /tmp/tmp.key \
  --cert-path /tmp/tmp.cert

This was all incredibly annoying. Annoying enough that it warrants recording it here for posterity for my own selfish purposes, and possibly for the next poor soul trying to figure this out and maybe googling for something that brings this post up. If that’s you? Good luck. I hope this works.

Weeknotes for the week ending Mar.4

04 Mar 2022

"Middle of nowhere" Utah, mid-x-country drive back to NE from San Diego - November 2001.

I’m certain I’ve told this story before to many, many people. But I don’t care - I’ll tell it anyway.

Almost 21 years ago I was working at a small-ish web development agency (RIP Screenhouse) and the bottom was just about to start falling out from beneath the “dot com” industry. Easy come, easy go. Things started to dry up, and there were quite a few developers and designers just, kinda, twiddling their thumbs. Some folks got together and started designing and coding an internal intranet with hopes that maybe it could be sold, licensed out, etc. SaaS but … pre-SaaS. That was alright, I guess? But I felt like doing something more fun. Remember, early-ish days of the internet – pre-social media. So, of course, I started hacking together a messageboard with PHP and Mysql. It was an opportunity to try some new tech in a sandbox that my friends and I could mess around with/in.

Fast-forward 15, 16 years later and I had continued to work on it in some form or another - 2 rewrites in PHP, 3 rewrites in Ruby/Rails. By that time I slowed down, getting along in my career, started a family, having less time and enthusiasm for maintaining the code. I’d managed to extract the bulk of the code into an open source ruby gem and was relatively happy with where it was. So it was a fortunate turn of events that Gleb, this brilliant engineer in London, got in touch with me and started contributing (a LOT!) to the codebase. Another guy, Tim, also came along and helped push things to another level.

Which brings me to why I mention this now. This week Tim and Gleb got the Thredded project to a 1.0 release, which is a huge milestone considering the modest beginnings of the project. Twenty-one years after a very immature, very green, very stupid, kid started slinging awful PHP as a means to connect with his friends and act as a test-bed for learning new technology 1

I’m genuinely shocked that it’s managed to kick around this long. As much as I am shocked, I am doubly thankful for the stewardship, care, and incredible work that Gleb and Tim have put into the project. I may have planted a seed 21 years ago, but the two of them have tended to, grown, the project into a remarkable gem of a gem. It’s way more their accomplishment at this point than it is mine.

So to the two of them – Gleb and Tim, congratulations on shipping Thredded 1.0. Cheers 🥂

1: In many ways I have that project, and the thousands of hours I spent on it, to thank for being the test-bed for the experimentation and learning that got me to where I am in my career today.

Weeknotes for the week ending Feb.25

25 Feb 2022

Obligatory trip to our favorite neighborhood park to enjoy the fresh slush snow

Did you know that Docker will automatically write firewall rules, using iptables, so that whatever ports you expose to the host OS will be allowed visibility out to the world?

I didn’t.

At least until this week.

It makes sense, I suppose? There’s a lot going on under the hood between the OS and network interfaces so making sure all of that plumbing is wired up sufficiently is a sane assumption to make. All that being said, you’d think that ufw deny <whatever port> would supersede and take precedence over anything Docker was doing on its own. Such is not the case. After spinning up an instance of PostgreSQL on a test Linux box using docker, mapping the container’s port to the host with:

  - "5432:5432"

… running ufw deny 5432; ufw reload, you might think that port 5432 would not be exposed to the world. If so, why would I be writing this? :guy-tapping-head-meme:

Turns out this is a common enough problem where cursory google searches return results like this.

UFW is a popular iptables front end on Ubuntu that makes it easy to manage firewall rules. But when Docker is installed, Docker bypass the UFW rules and the published ports can be accessed from outside.

Luckily I remembered that there may be an easier way (spoiler: there is) and fixed my problem with the following – why not bind to only the loopback interface: localhost aka A-HA! ☝️

Et voilà:

  - ""

There you go. A port scan with nmap will now result in your postgres server no longer being exposed to the internet.

Let that be a lesson to you, kids. Always check your own shit, lest your weakly password’ed databases be prone to pwn.


  • Listening: I hate that it’s exclusive to Spotify but these days I imagine you’re in the minority if you’re not streaming over there. I digress. This episode of “No Skips” about Ready to Die is outstanding. So much that I’d never taken a moment to notice, is dissected and pointed out and enumerated on. They say that the music you listen to as a teenager is what will stick with you for your entire life, and it’s true. Something said about Juicy on this episode really struck me. That song has been played so much over the last 27+ years that it has no business being so listenable today. So many popular old songs get to the “ok that’s enough. please, no more” stage … but, Juicy? No. “Yeah. This album is dedicated to all the teachers that told me I’d never amount to nothin’” … keep it going. Hall of fame track by the greatest to ever do it.
  • Watching: The Tourist, a show out of Australia, by the BBC, with a Scottish lead. The premise is good (quite Memento’ish?) and the first few episodes deliver, but once everything is tied together it leaves a bit to be desired. I enjoyed it. Sara lost interest about 4 episodes in. Also watching Inventing Anna which is … not my thing.
  • 🇧🇷 Learning: “Vaso ruim não quebra”. Literal translation: “bad vases don’t break”. Meaning: “that piece of shit guy will outlive us all”. See: current world affairs. See, also: #45.
Joel Oliveira

Hi, I'm Joel.

An engineering manager in Boston. Lapsed "tweeter" at @jayroh. Thinking about family, my (bull)dogs, code, comics, and building things like shubox.

Articles from 2012

Articles from 2007