Introduction
Scratchwork.xyz is a website I made displaying the last 7 days of exterior temperature/humidity, compared with temperature/humidity measurements collected inside my house. The chart is created using Plotly in Python, and the webserver also hosts a simple REST API with basic authentication, letting an ESP32 microcontroller submit its collected data into an SQL database.
I first started working on this project several weeks prior to Georgia Tech shutting down campus for COVID-19 precautions. I finished a few days prior to the shutdown, which was bad timing – I had little left to do but tweak it, but far more free time without a commute. Unfortunately my timing was off for a second reason: the server that hosted the website, SQL server, and REST API used Ubuntu 18.04 which was about to be replaced by 20.04. Rather than upgrade directly now that 20.04.1 is available, I decided to re-create the project and write up the process as if creating it for the first time.
Along the way, if I have something interesting but not imperative, I'll put it over in a sidenote.I had some background in Python, Linux, and microcontrollers prior to beginning, but almost no experience with web-based technologies like REST APIs or website hosting/serving. Even the simplest tasks took hours to debug. When I got stuck, I tried to break down what I was working on into even smaller steps, so that each single step was simple enough to be solved by a single tutorial or documentation entry, even as a beginner. I will try to point out various tutorials that were helpful for different steps in this project. I also found it helpful to create several intermediate states when the next step seemed too confusing, which both solved and created various sticking points. One intermediate testing step took days to troubleshoot, and didn't even have any carryover to the final project!
The overall framework is loosely based on a description I found of the AdafruitIO service, which is itself a wonderful way to get started in remote sensing/IoT projects without messing with your own server. I chose to keep as much of the coding as I could in Python, since I was familiar with it. I even used Micropython on the ESP32! A goal of this project was to understand all the steps required to create a simple data collection and analysis platform. Although I just use it to compare outside weather (collected using a 3rd-party API) with my own indoor climate, the basic framework could be used for many other purposes.
I’m going to include intermedite steps that aren’t strictly necessary, which will allow you to follow the thought process in small steps as we build up a useful service. I will try to be explicit in all my reasonings without going into boring detail (especially if the details are provided in tutorials I link), but it might be hard to follow along if you’ve never used Linux or Python at all. Luckily, they’re both free to use and learn: I highly recommend installing Linux on a personal computer alongside your main operating system, so you can get used to the way it’s organized. Use Ubuntu if you’re a beginner (I prefer the Mate desktop environment version). The server will use Ubuntu, but it won’t have a desktop environment: everything will be done in the command line. But remoting into it from a Linux desktop of your own is a breeze.
I’ll make a Github repository of example files used on the server for the various services we are going to set up. If you try to follow along and get super-stuck, please reach out. I definitely don’t know everything but this project helped hone my troubleshooting abilities immensely! ;-)
Ok… ready to get started?