=========================================================================== ==+== Reviewer: Bryan Ford ==-== Title: Live Streaming Performance of the Foovid Network ==-== Date: 9-Jun-09 ==+== ===================================================================== ==+== Scoring (1-5) ==+== ==+== A. Overall merit: ==-== Overall paper merit ==-== Choices: 1. Bottom 50% of submitted papers ==-== 2. Top 50% but not top 25% of submitted papers ==-== 3. Top 25% but not top 10% of submitted papers ==-== 4. Top 10% but not top 5% of submitted papers ==-== 5. Top 5% of submitted papers! ==-== Enter the number of your choice: 3 ==+== B. Reviewer qualification ==-== Your qualifications to review this paper ==-== Choices: 1. I know nothing about this area ==-== 2. I have passing familiarity ==-== 3. I know the material, but am not an expert ==-== 4. I know a lot about this area ==-== 5. I am an expert on this topic ==-== Enter the number of your choice: 3 ==+== C. Novelty ==-== Choices: 1. Published before ==-== 2. Done before (not necessarily published) ==-== 3. Incremental improvement ==-== 4. New contribution ==-== 5. Surprisingly new contribution ==-== Enter the number of your choice: 3 ==+== D. Paper summary ==-== Please summarize this paper in your own words. This paper describes the architecture of Foovid, a "peer-assisted" live video streaming system under commercial deployment, and measures its performance under real use, including during a time period when Foovid was used to stream the UEFA European Football Championship to up to 60,000 clients. The architecture is based on Peer-Division Multiplexing, slicing up the video stream into ECC-redundant substreams and sending each substream concurrently on a separate P2P overlay to minimize the effect of failures, delays, and jitter in the P2P redistribution network. The measurement study uses data collected both from the server and from the participating clients, the latter in particular enabling measurement of delay, errors, and jitter (approximately) as perceived by the users. ==+== E. Exciting ==-== What is exciting about this paper? Be brief, please. The paper provides a fairly in-depth analysis of a live streaming system in real use, and the delay measurements based on client-side data are nice. ==+== F. Boring ==-== What is boring about this paper? Be brief, please. My main complaint about the paper is that while it does seem to do a good job at validating the effectiveness of a particular live streaming architecture, it leaves me unsure what this paper teaches us about the Internet in general, beyond this specific streaming application. It's a good paper but it feels a bit more like a "how to design a good live streaming system" paper, which might fit better in NSDI for example, than an Internet measurement study. ==+== G. Detailed comments ==-== Specicific technical and/or presentation comments on the paper. The paper spends a bunch of time in the architecture section talking about how retransmission works, only to say somewhere else that retransmission is not actually used in the current deployment; if this is the case the description of the retransmission mechanism seems unnecessary. Especially without any kind of corresponding performance analysis in the results sections. More generally, the architecture section has a lot of detail that might be interesting in a "live streaming architecture" paper but feels out of place in an Internet measurement paper. The rather low measured "sharing ratio" of 0.2-0.35, indicating how much P2P clients were able to "help" the streaming servers, actually sounds to me like a pretty strong negative result in terms of the practicality of peer-assisted live streaming versus just direct streaming from centrally-managed servers or CDNs. If the peer-assist is only providing a 20-35% performance boost, that may be significant, but is it justified against the complexity of the architecture, the potential risks of maliciously hacked clients disrupting the network, etc.? Is this low sharing ratio primarily a result of the pervasiveness of "bad" NAT, or of the limited uplink bandwidth of average home broadband connections, or of Foovid's goal of minimizing delay (and hence probably number of hops, especially through "marginal" peers), or all of the above, in some combination? I would have loved to see a lot more detailed discussion and analysis of this issue: e.g., can you estimate based on your data how much better/worse the sharing ratio would be under different network conditions, tighter/looser restrictions on user-observed performance, a larger ratio of users with "really high speed" (e.g., fiber to the home) connections, etc.? Can the architecture described be tweaked to operate at different tradeoffs or "sweet spots" between sharing ratio and perceived performance, depending for example on server resources available at a given time? The NAT traversal experiments in section 5 seem a bit simplistic and not very useful. Given that you can and do instrument the client hosts, it would be much more satisfatory if you could report and analyze "real-world" NAT traversal statistics based on the clients' actual attempts to establish P2P streaming connections with each other. Of course, I understand that this particular reporting functionality might not have been part of the clients when the data was collected, but it would be very nice to see such data in future traces - especially for their value in answering interesting questions like the ones above.