In recent years, given their rapid propagations, Internet worms increasingly threaten the Internet hosts and services. It's worsen by the fact that zero-day polymorphic worms, which can change their patterns dynamically, would evade most existing intrusion detection systems which depend on some signature generating approach. In this paper, we propose a novel rough set worm detection (RSWD) scheme which extends well developed rough set theory (RST) to detect zero-day polymorphic worms and provide a minimum set of filtering rules to network barrier equipments, such as firewall, to block worm spreading. The RSWD scheme is based on an assumption that, for a polymorphic worm, all attack packets are generated from some specific worm program and attack the same vulnerability of the victim hosts, therefore some patterns exist even the polymorphic engine mutates dynamically and frequently. Our simulations show that, in a class B network containing a new polymorphic worm which can not be recognized by any known signature, the RSWD module could detect the worm propagation within 17 s and produce a precise blocking rule exhibiting 100% true positive rate and 99.82% accuracy rate. (C) 2008 Elsevier Ltd. All rights reserved.
