Winter + Summer 2016
Adknown is a high-performance advertising company that connects customers to advertisers. Their business model is based on targeting users that are likely to be interested in their advertiser’s products. To enhance ad targeting they use various Big Data Analytic methods to analyze billions of user requests at a time, searching for patterns and trends.
My time at Adknown was spent managing the Mobile Real Time Bidding platform, RTB for short. The project dealt with interacting with ad exchanges that would hold auctions for ad space. We would bid in these auctions by getting ‘bid requests’ from exchanges, and reply back with a ‘bid response’. The response would either be a ‘nobid’ response, or a bid response containing our bid value and a link to our ad if we had the highest bid at the auction. The entire process of receiving the request, determining if we were to bid on it and replying back with a response took a fraction of a second, and we would be getting anywhere from 50 million to 200 million requests a day. Each request and response would be stored using Amazon’s Simple Storage Server (S3).
Managing RTB involved prioritizing and completing tasks that would be created during the day to day operations of the platform. Tasks could be created by any actor involved in the project (i.e. myself as the project manager, the SEM using the platform as a User, my boss as the IT Director). This meant deciding when and how new features were to be implemented, what kind of safeguards and diagnostic tools we needed and fixing any bugs or problems that would appear. It was a challenge initially to wrap my head around prioritizing the tasks and balancing my time between tasks, but it came together.
To complete the tasks and goals set forth by RTB, various technologies were utilized. Our backend for various servers was written in PHP, while the frontend for any User Interfaces to the platform was HTML5, PHP, CSS3 and Javascript, utilizing the bootstrap and JQuery frameworks. Our Server back ends ran on Nginx, and utilized Redis for in-memory data lookup and FluentD for request, response and error logging. As the primary developer on the project, I had to know how all of these technologies worked and how they worked together. It was a steep learning curve, but I was able to pick them up and was semi-frequently consulted by full time developers on the nuances of the different services.
RTB utilized Amazon’s Web Services platform for our server and data storage needs. This meant there was no physical racking of servers on our end – with a few key strokes we could go from having two servers to 10. It removed a lot of the traditional headache associated with adding and removing servers. It was also part of my job to manage the AWS platform and minimize costs wherever I could. This meant optimizing the number of servers we used for bidding, ensuring we were storing our data efficiently, integrating new services that would benefit the project and monitoring the platform to ensure we were still bidding and making money. Monitoring the platform was fairly complicated at first as there were many reasons for us to stop bidding. Tracking down the issue wasn’t always easy, and usually entailed examining and code over multiple servers. I did manage to standardize the process and leave a troubleshooting guide for the developer that was taking over the project once I left.
The major services we used of the AWS platform were as follows:
- EC2 Auto Scaling – This is how we could automatically provision more servers if the load coming in from our partnered exchanges ever became too great. New servers would automatically be booted up and added to the list load balancing tool
- EC2 Elastic Load Balancer – A service for sending server requests to multiple different copies of servers. If we had three servers- A, B and C – and we had 10000 requests in a single second, the ELB would distribute the requests such at that A might get 4,000 requests, B might get 4,000 requests and C would get the remaining 2,000.
- Code Deploy – A service for deploying code changes to groups of servers. At any point, we could have between 2 and 100 different bidding servers. If we made a change to the bidder code, it would be annoying and time consuming to deploy the changes to the servers one at a time. With code deploy, it automates the process. You can even decide to deploy the changes to all the servers at once if you so felt like it.
- Code Pipeline – An advanced Code Deploy alternative that allows you to set up staging an unit tests. This service would call Code Deploy to deploy the changes and then use an AWS Lambda Function to run our unit tests for the bidder. Should anything fail, I would be notified immediately and I could revert the change while a patch was made.
- Simple Storage Service (S3) – This is how we stored our data. We would keep our ads in S3, as well as store our requests and responses here. Much cheaper than maintaining our own data storage unit.
- Redshift – A data warehouse service. We would load in data from S3 and be able to analyze billions of requests at once using PostgreSQL queries, allowing us to look for trends to increase profits.
- Relational Database Service (RDS) – A wrapper over their Ec2 instances that would house many different types of relational database models. At Adknown, we mostly used MySQL
- Elasticache – Hosting service for various in-memory data stores and caches. We used it for creating master-slave Redis networks.
- AWSLogs – A congregation of access and error logs for our servers. It meant you only needed to go to one spot to check the error logs of all servers instead of logging into each one.
My time managing RTB has shown me the power of the AWS platform, and it has swayed me in the direction of focusing on expanding my knowledge of its use.
Upon being hired at Adknown, I did not know what to expect of my next eight months. I had yet to fully grasp the opportunity I was given. At Adknown, they care a lot about training their people well and giving them opportunities to grow on their own. I learned a great deal while working for Adknown, but here are some of the more important points that I feel I should mention.
- Understanding the goals that the software is trying to achieve can be just as important, if not
more, than understanding all the different inner workings of the system.
It doesn’t take much to be able to take a set of requirements and implement them. You follow the requirements to a T and ask questions where the instructions are unclear or ambiguous. The problem with that is you now need someone who understands the purpose of the system to manage the requirements. This is costly in resources, and you can add so much value to a company if you understand the functional side of the system you’re building. By understanding the functional side, you can examine a feature and possibly come up with a better way of accomplishing the goal the requirement set out to solve. You may also see future problems with the feature, and implement it in such a way that fixing the future problem takes minimal work. Overall, understanding the functional side of the system makes you a better developer.
- It’s easy to fall into the trap of coming up with an overly complex solution.
Always try to keep in mind the goal of whatever code you’re writing. Revisiting the goals and analyzing if this extra thing is needed to achieve that goal will help in pushing out features. There were many times during the term where I would blindly try to future proof a feature and make it extendible, only to have the feature scrapped a month later because it didn’t do what we needed it to. I wouldn’t keep the goals of the feature in mind, and I’d lose my way in designing the implementation. Maintaining a focus on the goal is imperative when designing an implementation.
- When you’re trying to do something quick, there will always be a second release. If there isn’t a
second release, it means you didn’t cut enough corners and the bare minimum of the feature could
have been pushed out faster.
This mostly applies to proof-of-concept features, but none the less, it is a skill. In school, we are hammered with good-design principles and proper ways to build things. Some of these principles take more time to implement and over complicate the system with extendable hooks for later. If you’re trying to get something out at the absolute fastest possible time, you need to train yourself to ignore some of the principles you’ve been taught. Once the feature is out and it has been proven that we will want to keep it, then you go back and use the principles you ignored the first time around to do it right.
- Colleagues are a great resource in avoiding spinning wheels and taking too long to get something to
work.
A soft skill that just about everybody needs to develop is the ability to know what questions to ask and when to ask them. I’ve never had an issue with figuring out the right question to ask, but knowing when to ask has been something I’ve been working on. I tend to lean towards not asking a question quickly enough, and will spend too much time trying to figure out the solution to a problem on my own a colleague might already know. Steps I took to try to improve on this is to keep an internal clock on how long I was trying to solve a problem, and at the end would post in our ‘Dev Chat’ skype group.
Overall, a fantastic co-op experience working for Adknown. They treat and train their employees well and as such have retained a highly skilled workforce. They offer many avenues to grow and expand your skillset and the company culture is very conducive to learning and collaboration. I look forward to working with them again post-graduation.