7/31/2011

Baidu Tieba Architecture

An architect of Baidu, who is being in charge  of the technology of Tieba product, gave a brief introduction on the back end technologies of this famous web application in the June activity of Baidu Salon.
Here are some notes on this speech:

Part I – Application Scale

1. Not just simple plain forum, but also photo/video/gaming
2. Includes front end, storage, anti-spamming, searching and mining
3. Numeric facts
- Bs of topics
- 10Bs of posts
- 10Ms of posts for single hot topic
- Ps of video data
- 100K+ QPS from client web browser
- 10K+ per second update message forwarding (I doubt this number)
- 100s service

Part II – Backend Technology: lightweight framework

For 80% common situations
1 MySQL
- prefer InnoDB than MyIASM with some modification (on disk writing pattern, with 10x perf gain)
- application optimization:
* avoid joining (by break normalization?)
* auxiliary index
* data locality
- single node numbers
* Ks of QPS
* 100Gs of Data
- mySql clustering
* master/slave for write/read separation
* home brewed request dispatcher (for easier programming and load balancing)
2 Cache
- hit ratio around 80%
- 10k ~ 100k QPS
- multiple granularity (page, picture, data item etc.)
- challenge: cache updating, writing request pressure
3 Flash Disk
- 5x – 10x perf gain without extra effort
- huge improve on random access
- size limitation: 500G (SSD) vs 10T (HDD)

Part III – Backend Technology: Heavyweight Infrastructure

For 20% rare scenarios
1. Partitioning
- Virtual, partitioned by application
** topic and post are separated
** relationship(list) and content are separated
- Horizontal, partitioned by key
2. Message Queue
- Reliable multicast communication system
- Handling mutation requests (I guess)
- Peak tps:100K+ (really?)
- It can only solve updating reliability problem, but seems that the speaker claims it also solves the scalability problem
3. In house storage node
- speedup by transforming random write to batch/append write
- memory patch: (background merge [mem + disk] in my understanding)
- write ahead logging for reliability
- highly optimized for application
4. In house distributed KV store
- for video storage
- replication (driven by MQ) for reliability
- append only
- Peta bytes scale

Part IV Backend Technology – Clustering Management

1. Most website are basically SOA architecture
- 100+ standalone small services
- service orchestration for single user request
2. Challenges in this architecture
- service/data upgrading
- failure handling
- performance variation
3. Service Management
- service metadata center management
- service registration and notification
- hide service cluster from application caller
- auto failure handling and load balancing
- (why service notification but not just try and ask registry if failed?)

Part V – Summary

It seems that there is nothing new in the presentation, all related technologies are well known. But its value lies in the fact that it gave us a high level overview of how today’s various famous web service is implemented and many numeric facts about this product.

[Reference]

0. Baidu Salon site
1. Speaker introduction and video
2. Speech ppt

No comments: