Statistical Science

Network Tomography: Recent Developments

Rui Castro, Mark Coates, Gang Liang, Robert Nowak, and Bin Yu

Full-text: Open access

Abstract

Today’s Internet is a massive, distributed network which continues to explode in size as e-commerce and related activities grow. The heterogeneous and largely unregulated structure of the Internet renders tasks such as dynamic routing, optimized service provision, service level verification and detection of anomalous/malicious behavior extremely challenging. The problem is compounded by the fact that one cannot rely on the cooperation of individual servers and routers to aid in the collection of network traffic measurements vital for these tasks. In many ways, network monitoring and inference problems bear a strong resemblance to other “inverse problems” in which key aspects of a system are not directly observable. Familiar signal processing or statistical problems such as tomographic image reconstruction and phylogenetic tree identification have interesting connections to those arising in networking. This article introduces network tomography, a new field which we believe will benefit greatly from the wealth of statistical theory and algorithms. It focuses especially on recent developments in the field including the application of pseudo-likelihood methods and tree estimation formulations.

Article information

Source
Statist. Sci., Volume 19, Number 3 (2004), 499-517.

Dates
First available in Project Euclid: 16 March 2005

Permanent link to this document
https://projecteuclid.org/euclid.ss/1110999312

Digital Object Identifier
doi:10.1214/088342304000000422

Mathematical Reviews number (MathSciNet)
MR2185628

Zentralblatt MATH identifier
1100.62628

Keywords
Network tomography pseudo-likelihood topology identification tree estimation

Citation

Castro, Rui; Coates, Mark; Liang, Gang; Nowak, Robert; Yu, Bin. Network Tomography: Recent Developments. Statist. Sci. 19 (2004), no. 3, 499--517. doi:10.1214/088342304000000422. https://projecteuclid.org/euclid.ss/1110999312


Export citation

References

  • Berger, J. O., Liseo, B. and Wolpert, R. L. (1999). Integrated likelihood methods for eliminating nuisance parameters (with discussion). Statist. Sci. 14 1--28.
  • Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems (with discussion). J. Roy. Statist. Soc. Ser. B 36 192--236.
  • Besag, J. (1975). Statistical analysis of non-lattice data. The Statistician 24 179--195.
  • Bestavros, A., Byers, J. and Harfoush, K. (2002). Inference and labeling of metric-induced network topologies. In Proc. IEEE INFOCOM 2002 2 628--637. IEEE Press, New York.
  • Blackwell, D. (1973). Approximate normality of large products. Technical report, Dept. Statistics, Univ. California, Berkeley.
  • Cáceres, R., Duffield, N., Horowitz, J. and Towsley, D. (1999). Multicast-based inference of network-internal loss characteristics. IEEE Trans. Inform. Theory 45 2462--2480.
  • Cao, J., Davis, D.,Vander Wiel, S. and Yu, B. (2000a). Time-varying network tomography: Router link data. J. Amer. Statist. Assoc. 95 1063--1075.
  • Cao, J., Vander Wiel, S., Yu, B. and Zhu, Z. (2000b). A scalable method for estimating network traffic matrices. Technical report, Bell Labs.
  • Castro, R., Coates, M. and Nowak, R. (2004). Likelihood based hierarchical clustering. IEEE Trans. Signal Process. 52 2308--2321.
  • Chao, X., Miyazawa, M. and Pinedo, M. (1999). Queueing Networks: Customers, Signals and Product Form Solutions. Wiley, New York.
  • Coates, M., Castro, R., Nowak, R., Gadhiok, M., King, R. and Tsang, Y. (2002a). Maximum likelihood network topology identification from edge-based unicast measurements. In Proc. ACM SIGMETRICS 2002 11--20. ACM Press, New York.
  • Coates, M., Hero, A., Nowak, R. and Yu, B. (2002b). Internet tomography. IEEE Signal Processing Magazine 19(3) 47--65.
  • Coates, M. and Nowak, R. (2000). Network loss inference using unicast end-to-end measurement. In Proc. ITC Seminar on IP Traffic, Measurement and Modelling 28-1--28-9. Available at citeseer.ist.psu.edu/context/1699850/514748.
  • Coates, M. and Nowak, R. (2002). Sequential Monte Carlo inference of internal delays in nonstationary communication networks. IEEE Trans. Signal Process. 50 366--376.
  • Cox, D. R. (1975). Partial likelihood. Biometrika 62 269--276.
  • Csiszár, I. (1975). $I$-divergence geometry of probability distributions and minimization problems. Ann. Probab. 3 146--158.
  • Duffield, N., Horowitz, J. and Lo Presti, F. (2001). Adaptive multicast topology inference. In Proc. IEEE INFOCOM 2001 3 1636--1645. IEEE Press, New York.
  • Duffield, N., Horowitz, J., Lo Presti, F. and Towsley, D. (2002). Multicast topology inference from measured end-to-end loss. IEEE Trans. Inform. Theory 48 26--45.
  • Duffield, N., Lo Presti, F., Paxson, V. and Towsley, D. (2001). Inferring link loss using striped unicast probes. In Proc. IEEE INFOCOM 2001 2 915--923. IEEE Press, New York.
  • Fasulo, D. (1999). An analysis of recent work on clustering algorithms. Technical Report 01-03-02, Dept. Computer Science and Engineering, Univ. Washington. Available at citeseer. nj.nec.com/fasulo99analysi.html.
  • Harfoush, K., Bestavros, A. and Byers, J. (2000). Robust identification of shared losses using end-to-end unicast probes. In Proc. IEEE International Conference on Network Protocols 22--33. IEEE Press, New York. Errata available as Technical Report 2001-001, Dept. Computer Science, Boston Univ.
  • Hastings, W. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57 97--109.
  • Ihaka, R. and Gentleman, R. (1996). R: A language for data analysis and graphics. J. Comput. Graph. Statist. 5 299--314.
  • Kelly, F. P., Zachary, S. and Ziedins, I., eds. (1996). Stochastic Networks: Theory and Applications. Oxford Univ. Press.
  • Leland, W., Taqqu, M., Willinger, W. and Wilson, D. (1994). On the self-similar nature of Ethernet traffic. IEEE/ACM Transactions on Networking 2 1--15.
  • Liang, G. and Yu, B. (2003a). Maximum pseudo likelihood estimation in network tomography. IEEE Trans. Signal Process. 51 2043--2053.
  • Liang, G. and Yu, B. (2003b). Pseudo likelihood estimation in network tomography. In Proc. IEEE INFOCOM 2003 3 2101--2111. IEEE Press, New York.
  • Lo Presti, F., Duffield, N., Horowitz, J. and Towsley, D. (2002). Multicast-based inference of network-internal delay distributions. IEEE/ACM Transactions on Networking 10 761--775.
  • Morris, R. and Lin, D. (2000). Variance of aggregated web traffic. In Proc. IEEE INFOCOM 2000 1 360--366. IEEE Press, New York.
  • O'Sullivan, F. (1986). A statistical perspective on ill-posed inverse problems (with discussion). Statist. Sci. 1 502--527.
  • Padmanabhan, V. N., Qiu, L. and Wang, H. (2002). Passive network tomography using Bayesian inference. In Proc. ACM SIGCOMM Workshop on Internet Measurement 93--94. ACM Press, New York.
  • Pásztor, A. and Veitch, D. (2002). PC based precision timing without GPS. In Proc. ACM SIGMETRICS 2002 1--10. ACM Press, New York.
  • Ratnasamy, S. and McCanne, S. (1999). Inference of multicast routing trees and bottleneck bandwidths using end-to-end measurements. In Proc. IEEE INFOCOM 1999 1 353--360. IEEE Press, New York.
  • Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific, Singapore.
  • Robert, C. and Casella, G. (1999). Monte Carlo Statistical Methods. Springer, New York.
  • Rolls, D. (2003). Limit theorems and estimation for structural and aggregate teletramc models. Ph.D. dissertation, Queen's Univ., Kingston, Ontario, Canada.
  • Scott, D. (1992). Multivariate Density Estimation: Theory, Practice and Visualization. Wiley, New York.
  • Shih, M. and Hero, A. (2001). Unicast inference of network link delay distributions from edge measurements. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing 6 3421--3424. IEEE Press, New York.
  • Tebaldi, C. and West, M. (1998). Bayesian inference on network traffic using link count data (with discussion). J. Amer. Statist. Assoc. 93 557--576.
  • Tsang, Y., Coates, M. and Nowak, R. (2001). Passive network tomography using EM algorithms. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing 3 1469--1472. IEEE Press, New York.
  • Tsang, Y., Coates, M. J. and Nowak, R. (2003). Network delay tomography. IEEE Trans. Signal Process. 51 2125--2136.
  • Vanderbei, R. J. and Iannone, J. (1994). An EM approach to OD matrix estimation. Technical Report SOR 94-04, Princeton Univ.
  • Vardi, Y. (1996). Network tomography: Estimating source-destination traffic intensities from link data. J. Amer. Statist. Assoc. 91 365--377.
  • Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. J. Amer. Statist. Assoc. 58 236--245.
  • White, H. (1994). Estimation, Inference and Specification Analysis. Cambridge Univ. Press, New York.
  • Willet, P. (1988). Recent trends in hierarchical document clustering: A critical review. Information Processing and Management 24 577--597.
  • Ziotopolous, A., Hero, A. and Wasserman, K. (2001). Estimation of network link loss rates via chaining in multicast trees. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing 4 2517--2520. IEEE Press, New York.