July 25, 2010 — nischal
One fine saturday night, I sat down to develop JustUnfollow. I was developing it for myself. I have this itch of learning new stuff (especially if its from Google), so decided to use the Google Appengine.
It's been six months now and my feelings for the decisions I made are mixed. The appengine has its advantages as well as disadvantages. I've faced a lot of issues due to GAE's datastore coping with the high usage it gets from all the folks who have their apps running on it.??
Well, this post isn't about GAE's issues, I'm writing this for a different reason. When I developed JustUnfollow, I made use of sharded counters for stats. A few days back, I could see a lot of errors in my log and on inspecting realized the errors were all about DataStore writes. Maximum writes to the datastore were failing. I've disabled stats for now, but I'm trying to figure out what the problem is. The reason for failure is most likely the number of requests that are being made now. My shards need to be incremented around 5 to 8 times per second. I thought the sharded counters would be able to handle this, but seems like I was wrong.
What I'm going to do now is increase the number of shards from the current 25 to 100. I'm not sure if this is the right thing to do, if the number of shards I'm using are low, just right or too much. If any of you have any idea please let me know about it.