QTCP: Adaptive Congestion Control with Reinforcement Learning

• Published in: IEEE transactions on network science and engineering Vol. 6; no. 3; pp. 445 - 458
• Main Authors: Li, Wei, Zhou, Fan, Chowdhury, Kaushik Roy, Meleis, Waleed
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01-07-2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive control; Aerospace electronics; Algorithms; Bandwidth; Congestion; Delays; dynamic generalization; Encoding; Function approximation; Kanerva coding; Machine learning; Mapping; Optimization; Protocol (computers); Protocols; Reinforcement learning; TCP (protocol); TCP congestion control; Throughput; User satisfaction
• ISSN: 2327-4697; 2334-329X
• DOI: 10.1109/TNSE.2018.2835758

Next generation network access technologies and Internet applications have increased the challenge of providing satisfactory quality of experience for users with traditional congestion control protocols. Efforts on optimizing the performance of TCP by modifying the core congestion control method depending on specific network architectures or apps do not generalize well under a wide range of network scenarios. This limitation arises from the rule-based design principle, where the performance is linked to a pre-decided mapping between the observed state of the network to the corresponding actions. Therefore, these protocols are unable to adapt their behavior in new environments or learn from experience for better performance. We address this problem by integrating a reinforcement-based Q-learning framework with TCP design in our approach called QTCP. QTCP enables senders to gradually learn the optimal congestion control policy in an on-line manner. QTCP does not need hard-coded rules, and can therefore generalize to a variety of different networking scenarios. Moreover, we develop a generalized Kanerva coding function approximation algorithm, which reduces the computation complexity of value functions and the searchable size of the state space. We show that QTCP outperforms the traditional rule-based TCP by providing 59.5 percent higher throughput while maintaining low transmission latency.