-
The Service Analysis and Network Diagnosis DataPipeline
Authors:
Derek Weitzel,
Shawn McKee,
Brian Paul Bockelman,
John Thiltges,
Marian Babik,
Ilija Vukotic
Abstract:
Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements i…
▽ More
Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements into a message bus. The message bus allows the project to send the data to multiple consumers, including a tape archive, an Elasticsearch database, and a peer infrastructure at CERN. In this paper, we explain the architecture and evolution of the SAND data pipeline, the scale of the resulting dataset, and how it supports a wide variety of network analysis applications.
△ Less
Submitted 6 December, 2021;
originally announced December 2021.
-
WLCG Networks: Update on Monitoring and Analytics
Authors:
Marian Babik,
Shawn McKee,
Pedro Andrade,
Brian Paul Bockelman,
Robert Gardner,
Edgar Mauricio Fajardo Hernandez,
Edoardo Martelli,
Ilija Vukotic,
Derek Weitzel,
Marian Zvada
Abstract:
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It…
▽ More
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It was established to ensure sites and experiments can better understand and fix networking issues, while providing an analytics platform that aggregates network monitoring data with higher level workload and data trans-fer services. This has been facilitated by the global network of the perfSONAR instances that have been commissioned and are operated in collaboration with WLCG Network Throughput Working Group. An additional important updateis the inclusion of the newly funded NSF project SAND (Service Analytics and Network Diagnosis) which is focusing on network analytics. This paper describes the current state of the network measurement and analytics platform and summarizes the activities taken by the working group and our collaborators. This includes the progress being made in providing higher level analytics,alerting and alarming from the rich set of network metrics we are gathering.
△ Less
Submitted 1 July, 2020;
originally announced July 2020.
-
Network Capabilities for the HL-LHC Era
Authors:
Marian Babik,
Shawn McKee
Abstract:
High Energy Physics (HEP) experiments rely on the networks as one of the critical parts of their infrastructure both within the participating laboratories and sites as well as globally to interconnect the sites, data centers and experiments instrumentation. Network virtualisation and programmable networks are two key enablers that facilitate agile, fast and more economical network infrastructures…
▽ More
High Energy Physics (HEP) experiments rely on the networks as one of the critical parts of their infrastructure both within the participating laboratories and sites as well as globally to interconnect the sites, data centers and experiments instrumentation. Network virtualisation and programmable networks are two key enablers that facilitate agile, fast and more economical network infrastructures as well as service development, deployment and provisioning.Adoption of these technologies by HEP sites and experiments will allow them to design more scalable and robust networks while decreasing the overall cost and improving the effectiveness of the resource utilization.The primary challenge we currently face is ensuring that WLCG and its constituent collaborations will have the networking capabilities required to most effectively exploit LHC data for the lifetime of the LHC. In this paper we provide a high level summary of the HEPiX NFV Working Group report that explored some of the novel network capabilities that could potentially be deployment in time for HL-LHC.
△ Less
Submitted 1 July, 2020;
originally announced July 2020.
