Every coach and athlete that logs into Hudl immediately lands on their news feed. Each feed is tailored to each user and consists of content from teams they are in as well as accounts they choose to follow. This page is the first impression for our users and performance is critical. Our solution: ElastiCache for Redis.
Running sites with high availability is a foregone conclusion for most businesses. There are plenty of blog posts and articles out there talking about “nines”, but few really describe how to actually measure availability. Five nines of what, exactly? How do you measure continuous uptime of a website that serves discrete HTTP requests? Here’s how we measure server-side availability both overall for our individual microservices on hudl.com.
This post is a case-study about how we fixed some Redshift performance issues we started running in to as we had more and more people using it. A lot of the information I present is documented in Redshift’s best practices, but some of it isn’t easily findable or wasn’t obvious to us when we were just getting started. I hope that reading this post will save you some time if you are just getting started with Redshift and want to avoid some of the pitfalls that we ran into!
In August we migrated our core user data (around 5.5MM user records) from SQL Server to MongoDB. We moved the data during the daytime while still taking full production traffic, maintaining nearly 100% availability for reads and writes during the course of the migration. Our CPO fittingly described it as akin to “swapping out a couple of the plane’s engines while it’s flying at 10,000 feet.” I’d like to share our approach to the migration and some of the code we used to do it.
Over the last year, the Data Engineering squad has been building a data warehouse called Fulla. Recently, the squad rethought our entire data warehouse stack. We’ve now released Fulla v2 and Hudlies are querying data like never before giving us a better understanding of our customers and our product.
We took time to optimize our EC2 instance types. By finding the maximum load a server could handle we were able to run a quarter as many app servers. Our hourly spend dropped by 50%. Despite the huge cost savings, we also saw a 2x improvement in response times! This came about by moving to a newer instance family.
Hudl stores petabytes of video. In that video there are a lot of awesome plays. Figuring out which plays are the most interesting and sifting through the uninteresting footage is a huge challenge. To solve this problem, we leveraged deep learning, Amazon Mechanical Turk, and crowd noise. The result: basketball highlights!
One part of Hudl I frequently have to explain to people outside the company is the structure of our product team. Fellow developers at other companies, friends I graduated with, and plenty of people in between want to know how Hudl works—and as it turns out, there’s a lot to talk about. We’re constantly evolving and learning more about how to keep our heads on straight, and as we do, we want to get the lessons learned on the table.
Speed, innovation, and creativity… the key components of creative genius. Skunkworks is specifically designed to unleash the creative wrath of our product team. At Hudl, we use Skunkworks to explore new technologies and tools that make us better at what we do.
At Hudl, we like to move quickly. We are constantly fixing issues, building new features, and improving the experience for our coaches and athletes. We put a lot of thought into how we work and dedicate a lot of time to making sure we are working as efficiently as we can. So, when we began to run into major bottlenecks in our deployment process, we realized we needed a major change. We came up with a plan to break our monolithic application into smaller components, and thus The Multiverse was born.
As a company, we understand that one of our key competitive edges is moving quickly. We develop and ship new features continuously. Before we started moving toward our microapplication architecture, we were deploying our monolithic application ten times a day. Even though the number of Monolith deployments is trending down as we break it out into multiple, smaller applications, we still rely on our deployment infrastructure to deliver multiple payloads daily.
An autoscaling farm of AWS EC2 instances sits behind our front-facing web application, working on heavy, long-running tasks like video transcoding, thumbnail generation, and computer vision processing. It’s a battle-tested combination of queues, worker instances, and an orchestration service called Lifeguard that easily hammers through thousands of these CPU-bound jobs per minute.
Last year, we rolled out a new way for schools to raise money: Hudl campaigns. Online donation pages are nothing new to the web, so we went to the drawing board to find a way to make ours stand out. What we came up with was a compelling video with highlights from the team, great transitions and effects, and a heartfelt interview from the head coach. Find out how we automated this process and made something that took one person a few hours to manually craft into something that can be rendered automatically in around 10 minutes.
The self-learning path is the different from what most of us are used to, and perhaps a few tips from my journey (learned how to program 18 months ago without formal curriculum) may help you. These bits of wisdom were gained from hard-fought battles, but they were personal battles. A lot of these things will seem pretty obvious, some may seem over the top; if something doesn’t resonate with you, that’s fine. Take away what fits you.
When we began hacking away on Hudl we chose SQL Server as our database. After a few years and solid company growth we realized SQL Server was quickly becoming a bottleneck. Because we run in EC2, vertically scaling our DB was not a great option. That’s when we began to look at NoSQL seriously and specifically MongoDB. We wanted something that was fast, flexible and developer-friendly. In this post we’ll take a look at our schema design choices, our migration plan and the performance we’ve seen with MongoDB.
It’s not every day you’re asked to develop pages that will be viewed by millions of sports fans across the country during football season. But that’s exactly what Hudl did, and it’s nothing like an “intern project” at all.
At Hudl we use RabbitMQ to help us decouple some operations from impacting normal web requests. We smooth out spikes in write traffic and isolate long-running or CPU-intensive tasks. RabbitMQ is fast, full-featured, and offers good management/monitoring features.
Limiting yourself to one area of expertise introduces blind spots to your code. Learning several technologies across multiple layers in the stack makes you a better programmer, reduces bugs, and increases your value.
Logging in your application is super important. Early on it’ll be fine to peruse your logs manually. As traffic increases you’ll soon have a need to aggregate all that data into one place so it can be easily searched. At Hudl, we chose Splunk. There are a lot of competing products, but it works well for us.
Either way my advice is clear: Log. All the things. You won’t regret it.
There are thousands of Android devices on the market running 18 different versions of the Android OS. It’s impossible to test every combination of OS version, hardware, and screen size in house - but you have to face the reality that your app will run on all those configurations once it goes live. And you will run into unforeseen problems.
How do you manage the fragmentation problem and minimize the risk of problems when it goes live? I’ll give you a hint: it doesn’t need to be expensive or complicated if you use the right process.
Before coming to Hudl in January, I spent more than five years as a developer at Microsoft working on Dynamics AX. It was your very typical, old-school software product. However, in my time here at Hudl, I’ve been thrown into the world of Software as a Service (SaaS) and have come to absolutely love it. There are so many reasons why SaaS is better but here are my top five reasons why developers should switch to building SaaS products.