Solving portuguese elections problems with 10 EUROS…

January 23, 2011  |  cloud, dba, linux

Today is presidential elections day in Portugal. Every voter should know his/her voter ID and the location where he/she votes. The problem is that many people don’t remember their voterID and only remember to check it in the election day. Also, thousands of voters now own the new revolutionary portuguese Citizen Electronic Card and they aren’t aware of the changes taht this process made in their location voting address.

To help people find their voterID, some years ago, the MAI (Internal affairs) implemented a service for voters check their ID by submitting their CitizenID and birthdate using a website or sending an SMS to a service. This would be perfect, if it worked flawlessly in the election day…

But, it didn’t work as expected. Today the .NET based site was almost impossible to access and when users could open the site they were hitting lots of IIS Service Availability errors.

Portugal has 9.5 million estimated voters and I don’t believe that more than 5% of people tried this service. So in the worst case, 475000 people used this service. So lets imagine that 75% of voters used this service in the frametime between 11:00 and 15:00, the time most people vote. This would give around 360000 people checking up their voterID in 4 hours. As you probably know an hour is 3600 seconds, so at the highest peak 25 voters would check their VoterID every second. The question I do is: Is this too much? Is this a difficult issue to solve?

My answer… “WTF are these guys doing?” With a daily 10 EUROS infrastructure I would handle this problem and would serve all european voters at the same time! 🙂 How? Its simple… Check it out…

What we need to solve this problem?

1-Hardware infrastructure
1.1- Servers
1.2- Network connection

2-Software
1.1- A database with all voters IDs
1.2- An Http or APP server
1.3- A simple application

Now you ask, “how can you solve it with only 10 euros daily?” … My answer: “Dude, look to the sky… You can get over all that heat with the CLOUD!”. I would use cloud computing.

Let me explain… I would use 2 Amazon EC2 instances, one with a running VoltDB database and the other one with a python/php/java application running on an HTTP enabled web server.

As you know (or not), VoltDB is an in memory database, all the data rely on memory. So, how much memory would I need to store all this Voters info? Lets assume I would store 100 chars for the voter name, 10 chars for CitizenID, 50 for the location to vote, 30 chars for some other info, and finally 10 chars for the VoterID. Look, this would give 200 bytes! All portuguese voters basic info to solve this problem can be stored in 1.900.000.000 bytes, this is JUST 1.8 Gigabytes of memory! I would store all that info easily with an EC2 Large instance for the DB server and would run the application in a High CPU instance APP server. The instances I would choose would be:

Large Instance
7.5 GB memory with 4 EC2 Compute Units

High-CPU Medium Instance
1.7 GB of memory with 5 EC2 Compute Units

This hardware infrastructure would cost around 10 euros daily (yes, you can check AWS pricing to see that I’m right) and would solve the problems that didn’t allow lots of voters to do their civil rights!

If you read my previous post about VoltDB you remember that I easily processed around 13 thousands requests per second in my 5 years old IMAC. So with an EC2 instance like the one I describe earlier do you doubt that I would easily process 25 query requests per second? 🙂 The DB is not a problem… Neither the APP server isn’t it? 25 requests per second? Are you kidding? 🙂

Now you may say, well speaking “blah blah blah” is easy… Doing it, it’s a lot harder! And I say to you… I’ll do it and I will show it in a future post. I will load a 10 million records database with VoltDB, creating a simple python (or any other language) query application and use another instance to simulate the queries against this infrastructure… So keep tuned in and have a nice week…


20 Comments


  1. I’ll be keeping an eye in the blog for more details.

  2. Your math is wrong.

    “I don’t believe that more than 5% of people tried this service” – Number pulled from your ass.
    “So lets imagine that 75% of voters used this service in the frametime between 11:00 and 15:00” – Another number fallen from the sky

    “This would give around 360000 people checking up their voterID in 4 hours. (… an hour is 3600 seconds, so at the highest peak 25 voters would check their VoterID every second” – This is completely and utmost WRONG. You are assuming an uniform distribution for the requests and assuming that would be the peak. You should expect very high loads (tens of thousands?) after the URL for your service is mentioned in the media.

    • Additionally, do you think the PT gov. would (or should) trust amazon with their databases?

    • Chico Im not a statistics expert… And it wasn’t my intention to be. But what i can say is that with 100 euros I surely would serve easily more than 1000 requests per second using any cloud computing service available. But if you doubt, follow my blog in the next days and i will show what i Wrote here today.

      My post objective was to show that sometimes important services like this service are wrongly implemented and I wanted to show how easily they can be solved.

      You may not trust Cloud Computing if you dont understand how it works, but after you understand how your data can be encrypted, then you surely would trust the future… the future is cloud computing.

      regards
      Joao

    • Chico should he pull the numbers from your a**? Because it’s approaching the problem the best way, a pessimist educated guess.

      Edited by Joao (sorry JRA, I had to censor you word 🙂 )

  3. There’s a lot of buzz about security when using cloud computing. What I can say is that in my opinion the voters election data I talk here isn’t HIGH sensible data. Most of that information is already available on public gov sites. The problem is the trust or doubts about cloud computing services.

    What I can say is that lot’s of governments around the world are moving to cloud computing services, and most of the times they move data that’s much more sensitive (financial, criminal, etc) than the data my post talk about.

    I could point dozens of sensible public services using the cloud services… for instance US Treasury.

    http://www.yoursworld.com/u-s-treasury-moves-public-web-sites-to-amazon-ec2-cloud-1255.html

    At every breakthrough technology rise there are always doubts and concerns, but in the end… the technology will always prevail!

    Regards
    Joao

    • joao, you are overblowing it. None of the sites referenced by your link have confidential financial or criminal data. In fact not one offer services that require a personal account.

  4. Martinho Fernandes

    Nevermind your bad statistics.

    And let’s say you can really solve this with a 10€/day infrastructure. How much does the current infrastructure cost? Unless you have values to compare, you’re not proving anything at all.

    • Martinho, Have you read anywhere in my post that my approach is cheaper than the current used by the MAI?

      I only said that starting with 10 euros daily, some good programming and choosing the right technology I would create a better scalable service.

      Even with mysql memcached or a NOSQL db like Cassandra I would easily do a better job with few resources than the sluggish job made by those guys at MAI.

      If you doubt you can BET against me… 🙂 🙂 🙂 🙂 but beware 😉 , today in a couple of minutes I already put VOLTDB processing more than 3000 requests per second… And those results weren’t achieved with the approach “I am an excelent DBA!”… I got those results cause I’ve choose a SQL database that easily processes thousands query requests per second.

      Regards,
      Joao

  5. UPDATE 1:

    I’ve created a 10 million records table (1.5 GB) and started it with VOLTDB… I’ve processed several thousands query requests per second using a simple query java procedure. So, database bottleneck won’t be a problem. Tomorrow or the day after I’ll take care of the APP server part.

    Regards
    Joao

  6. I think you miss one obvious thing: the government needs to trust in your solution and test it beforehand.
    So, it will take millions of man-days, or hordes of highly paid managers to build the charts and presentations.
    In this economical situation Portugal has, you better stick what you have, the old solution they trust.

    It is the same with me, I lost interest towards new electronic devices like Ipad, HTC Heroes and etc. after trying them, and understanding they need long way to go…

  7. Solution PART I: Creating the database and loading it up with some data… i mean, 10 million records 😛

    http://www.beyondoracle.com/2011/01/29/10-million-records-voltdb-table/

    Regards
    Joao

  8. Hi there, You have done a great job. I will certainly digg it and
    personally suggest to my friends. I’m sure they’ll be benefited from this website.

Trackbacks

  1. Tweets that mention Solving portuguese elections with 10 EUROS… | Beyond Oracle -- Topsy.com

Leave a Reply