Shaun Heffernan SHAUN HEFFERNAN

We had 360 labeled clips of “Go Gators” in a variety of background noises, with the rest of the dataset hidden for testing. That is nowhere near enough to train a detector from scratch, so I wanted to try transfer learning. I found a dataset from TensorFlow, Google Speech Commands, that has a lot of command words which are similar to detecting wake words. We trained a convolutional encoder on it, attached a classifier head, and fine tuned the whole network on our recordings.

Each 3 second window becomes an 80 bin mel spectrogram, which better approximates human speech frequencies by compressing high frequencies and expanding lower ones. Three convolution blocks with filters doubling from 32 to 128 feed a dense layer of just 32, because we want the final prediction to not be made off of a lot of noise. That dense layer is surrounded by dropouts of 0.5 since we have such a small training set, and the final layer has 1 output with sigmoid: above 0.5 = wake word.

The first problem I ran into was the audio samples being 16 kHz instead of 48 kHz like our recorded data. I originally upsampled the Google speech commands to all be 48 kHz, but it seemed my upsampling was just adding noise, and validation F1 stayed in the 60s. So I tried downsampling the wake words to 16 kHz so they could match the Google libraries. Fine tuning with the encoder weights unfrozen performed significantly better than frozen, reaching F1 0.899 on the held out test set (Fig. 01b tells that story).

Everything in Fig. 01 is the real network: the demo clips are real recordings with precomputed scores, and “Record 3 s” scores your own attempt with the exported ONNX model in your browser. Audio never leaves your device. Write-up in papers ↓.

For dementia patients, the hardest part of a conversation is often who am I talking to? I'm on a UF Real World Engineering team building AI smart glasses that answer that in real time. The glasses stream camera frames and microphone audio over UDP to a Python backend where face recognition, voice activity detection, speech to text, and speaker diarization all run in parallel worker processes, and a coordinator fuses face and voice into one identity.

I built the audio pipeline core: 16 kHz audio in 160 ms chunks through Silero VAD, Parakeet streaming transcription on Apple MLX, and one ReDimNet voice embedding per sentence, matched by cosine similarity with a bias toward whoever spoke last. Benchmarked on the production path, transcription runs at RTF 0.085, about 12× faster than real time. When an unknown voice lines up with a known face, the system registers it and learns the voice on its own.

I also built the monitoring dashboard end to end: an aiohttp server that streams every worker's logs over a WebSocket into a live control panel with the identity database, per person chat history, and a face lookup tool. Plus the LLM agent worker that parses intent from speech, registers people when they introduce themselves, and logs every recognized sentence to a per person history.

Fig. 02 is a real run of this pipeline, not a mockup, recorded with teammates. Team repo on GitHub ↗

Pairing mentors with mentees is a two sided problem: a good match needs both people to actually want it. Mentor-Match has mentors and mentees rank each other, then runs a Gale–Shapley inspired matching round. Mentors propose, mentees hold on to their best offer, and a displaced mentor moves down their list until the whole pairing is stable and nobody would rather swap.

It shipped as a full product: a MERN stack web app with an Expo mobile app on top. Against hand-curated pairings the matcher reached ~80% match accuracy.

Fig. 03 animates one matching round from the algorithm. Run it and watch a displacement happen live.

The snooze button is a lie you tell yourself at 6 a.m. BioClock kills it: an ESP32 wearable whose on-device classifier reads motion and heart rate and only silences the alarm when it detects actual wakefulness.

Edge inference matters here: no cloud round-trip at the bedside, no raw biometrics leaving the device. Presented at the Warren B. Nelms IoT Conference. Drag the slider to feed the classifier.

Our data set is the NYC 2023 taxi cab data set. Each trip record captures pick-up and drop-off times, taxi zone locations, trip distances, itemized fares, rate types, and driver-reported passenger counts. We want to figure out what the most important factors are in predicting tip prices, to help taxi drivers optimize the amount of tips they collect.

Upon inspection of the dataset, only credit card tips are recorded and cash tips are not logged at all. We can't train a model to predict a value that we do not know, so the cash transactions had to be removed, roughly 20% of the nearly 10,000 trip dataset. I created a pre tip total by combining all of the costs that make up the total amount before factoring in the tip, and dropped the total amount because it includes the tip amount so there would be data leakage.

I trained linear regression with cross validation of 10 folds, and for the lasso a grid search picking λ from 0.0001 to 1 chose λ = 0.01. They had very similar R², 0.6033 vs 0.6075, but lasso dropped 15 of 27 features with a 0 coefficient: ride duration, passenger count, and most of the pickup days and times. For the taxi driver perspective, they shouldn't focus on the time of day despite the EDA suggesting otherwise. They should look for long rides, airport rides, and routes that pass through tolls because these are the best predictors for the tip amount.

Fig. 05 plots the real fitted coefficients from the report. Toggle LASSO and watch it prune. Write-up in papers ↓.