Close

The Google Appengine DataStore and troubles I’m facing

One fine saturday night, I sat down to develop JustUnfollow. I was developing it for myself. I have this itch of learning new stuff (especially if its from Google), so decided to use the Google Appengine.

It's been six months now and my feelings for the decisions I made are mixed. The appengine has its advantages as well as disadvantages. I've faced a lot of issues due to GAE's datastore coping with the high usage it gets from all the folks who have their apps running on it.??

Well, this post isn't about GAE's issues, I'm writing this for a different reason. When I developed JustUnfollow, I made use of sharded counters for stats. A few days back, I could see a lot of errors in my log and on inspecting realized the errors were all about DataStore writes. Maximum writes to the datastore were failing. I've disabled stats for now, but I'm trying to figure out what the problem is. The reason for failure is most likely the number of requests that are being made now. My shards need to be incremented around 5 to 8 times per second. I thought the sharded counters would be able to handle this, but seems like I was wrong.

What I'm going to do now is increase the number of shards from the current 25 to 100. I'm not sure if this is the right thing to do, if the number of shards I'm using are low, just right or too much. If any of you have any idea please let me know about it.

-Nischal

About the Author

  • arguin_app

    Nischal I think 25 shards are already too much. I am not sure about the amount of data that you are writing in the store, but the number is already too big, but the problem is the table remain locked for long time while you do write operation. I think you should consider reviewing your data indexes before increase the number. It is often critical to have a proper indexing strategy to achieve performance in a Key Value DB.Watch some google I/O videos for reference…

  • Anonymous

    @arguin_app I have to update this post. The issues were due to the datastore problem that the appengine was having. It’s running smooth now with the 25 shards :)