A few weeks ago, research was revealed that almost every computer chip built in the last 20 years contains fundamental security flaws. They were called Meltdown and Spectre and can steal data which is currently processed on a computer. Here at Studiosity, we ensure all data is kept secure. How? Our DevOps Engineer Lee Webb shares all the technological details in this post.
Firstly let me introduce myself.
My name is Lee Webb and I have been a DevOps Engineer with Studiosity now for almost 3 years. While the business title I carry has changed from role to role, I have always directly been involved with architecting, engineering and maintaining internet-facing computer systems and keeping them secure almost since I left High School. I find this interesting because the genesis of the Spectre and Meltdown exploits began when I left High School, way back in 1995 when a paper was released covering the rather drab topic of "covert timing channel in CPU cache and translation lookaside buffer (TLB) in the 80x86 Processor Architecture."
Fast forward some 23 years later and we finally have an exploit proof of concept, for what is basically the result of an architectural design decision which probably came about back in 1978 ... that's when I was born!
There's a theme here though, and it is 'time'.
Over the years, I've been working as a Systems Engineer there have been massive changes in the way in which we approach & use various tools to build and maintain computer systems and their security. Where, once upon a time a single computer in its physical box was curated and loved 'by hand', by teams of people, now it is just software, (a build) you install, according to a set of rules, then use and discard. The next time you need it you just 'build' it again.
Here at Studiosity I'm pretty proud of the fact that we 'build and destroy' computers (software wise) all the time, basically all day every day, it's just something which is part of modern IT system design, which everyone calls "the Cloud."
Why is this important?
Well, despite how many systems we actually have and what they are all doing at any moment, the mass application of patches or mitigations against vulnerabilities really isn't something which takes very much effort to do anymore, nor is there really much direct impact on the customer when we do it. It's just something that happens as part of normal operations. When there's a vulnerability, we address it and move on.
As soon as the email from Ubuntu Security Notifications arrived on Wednesday, I had a build plan authored and a Pull Request made within Github which was subsequently built, tested, then approved and deployed.
That afternoon we attended a company meeting while our instances within Amazon Web Services were modified and restarted, or simply terminated and replaced. This is essentially just a normal day in the life of our application.
Like most people, I found the notification from US-CERT on the 5th January seriously disturbing, and while the implications of an actual exploit of the vulnerabilities aren't a lot of fun to consider, I still felt confident that our current design and processes were sufficient to mitigate us against a platform exploit and release of customer data.
Here are some of the many reasons why I feel this way:
- Jack Goodman, our founder, was emailing people in the organisation the day before the US-CERT advisory, meaning that all of our management team were aware of the potential of this threat and were (and always are) on board when it comes to addressing platform issues, whatever they are.
- We deploy on Amazon Web Services, who take their security, and ours, very seriously.
- Amazon advised us that at the hypervisor level, they weren't vulnerable to this exploit. So I wasn't concerned about other AWS customers seeing our data.
- Our instances live inside a Virtual Private Network with enforced Network Traffic ACL's and Security Groups which only allow known authorised traffic between our machines.
- We run Transport Encryption everywhere with strong ciphers and protocols.
- All 'at rest' data is encrypted using tooling like OpenSSL / GnuPG or leverages AWS provided services such as S3 or EBS volume encryption to mitigate against physical data theft and release.
- Direct administrative access to EC2 instance consoles and SSH are tightly controlled to only a very small group of people who actually need that access (the concept of 'least privilege').
- We run network IDS processes at key ingress and egress points to identify possible network or system intrusion based on the network traffic.
- Logging and monitoring is constantly looking for error conditions and unauthorised access attempts (even this morning I blacklisted a few network ranges who were attempting to brute force their way into our VPN Bastion hosts, but to no avail).
- Each machine we build only runs the processes required by it's role, nothing more, and the instances themselves only have enough CPU and memory for what they do, so running unauthorised applications will generate an alert immediately.
- A lot of the Instances we run are purely stateless with very short lifecycles, built by our Continuous Delivery System as images and deployed on demand.
- Access to our application is through proxies, the application hosts themselves don't have a direct presence on the internet.
All of the little bits above (and more) basically mean that there was a very low probability of someone actually getting a foothold into our systems until vendor patches became available. That's why I felt and continue to feel confident about the security and privacy of our data.
To this end we're also working in the background to be compliant with the upcoming GDPR enforcement for EU citizens, something which our presence in the UK requires, and something which only benefits everyone else using the service, due to the increased privacy requirements.
While I don't help students directly, I know that the part that I play, and all the little things I do in keeping customer, and more importantly, student data private and secure help achieve amazing educational outcomes for all students who use our service.
