Endace Ltd is a privately owned network monitoring company, based in New Zealand and founded in 2001.[1] It provides network visibility and network recording products to large organizations. The company was listed on the London Stock Exchange in 2005 and then delisted in 2013 when it was acquired by Emulex.[2] In 2016 Endace was spun out of Emulex and is currently a private company.[3]
In October 2016, The Intercept revealed that some Endace clients were intelligence agencies, including the British GCHQ (known for conducting massive surveillance on network communications) and the Moroccan DGST, likewise known for mass surveillance of its citizens.
Background and history
Endace was founded after the DAG project at the School of Computing and Mathematical Sciences at the University of Waikato in New Zealand.[1][4] The first cards designed at the university were intended to measure latency in ATM networks.[5]
In 2006, Endace transitioned from component manufacturer to appliance manufacturer to managed infrastructure provider. The company now sells network visibility fabrics, based on its range of network recorders, to large corporations and government agencies.[6]
Endace was the first New Zealand company to list on London's Alternative Investment Market when it floated in mid-June 2005[7] a move which was not without controversy.[8] Poor share price performance in the early years and a seeming failure to attract a broad enough shareholder base lent weight to the criticism that Endace should have focused initially on developing its local profile (via NZX) rather than pushing for overseas investment (via London AIM).
The DAG project grew from academic research at Waikato University. Having found that software measurements of ATM cells (or packets) were unsatisfactory, both for reasons of accuracy and lack of certainty about packet loss, the research group set about developing their own hardware to generate better quality recordings.[5] This hardware and its subsequent iterations introduced two fundamental innovations: hardware timestamping and hardware accounting for packet loss.
Hardware timestamping
Conventionally, each packet or cell is given a timestamp by the host machine's kernel (i.e. in software) when the kernel driver is notified that a new packet has arrived. This approach results in poor quality timestamps for several reasons, among them the considerable latency and jitter between the packet arriving at the network interface and receipt by the kernel driver and uncertainty caused by interrupt coalescing wherein one host interrupt signifies the arrival of several packets. Such poor quality limits what research can usefully be done on network performance and related fields.
To solve this, the DAG generates timestamps in the hardware as close to the network interface as possible. Not only does this obviate latency, jitter and problems caused by interrupt coalescing, the hardware is capable of much greater accuracy and precision than software-generated timestamps. Precision comes from the freedom of custom hardware to assign as many bits to the timestamp as required and accuracy is assured by reference to an external time source such as GPS which is accurate to ± 40 nanoseconds.[9] In contrast, the accuracy of NTP (by which kernel clocks can be corrected over the Internet) is in the order of milliseconds (about 100,000 times less accurate), depending on the conditions involved.
The DAG produces 64 bit timestamps in fixed-point format with 32 fractional bits, giving a potential precision of seconds or 233 picoseconds. The actual precision offered varies with the particular model of DAG, the oldest giving 24 fractional bits (60 nanoseconds) and better precisions offered in DAGs for higher bandwidth networks.[10]
The timestamp is derived from a free-running clock provided by a crystal oscillator but the accuracy of crystals drift with both temperature and age. The DAG's solution is to use direct digital synthesis using the 1 Hz pulse-per-second output that many GPS receivers provide as its reference clock. This mechanism is described in §5.5.3 of Stephen Donnelly's PhD thesis[11] which also describes in detail the pre-commercial era models of DAG.
Crucially, and an academically significant contribution of the DAG, the ability to use an external reference such as globally synchronised GPS makes it possible to do one-way time-of-flight measurements. This is of immense interest to academic researchers because packets flowing between two points on the Internet are neither guaranteed to follow the same path in each direction nor guaranteed to have the same timing characteristics in each direction.
Almost as important as timestamp accuracy is guaranteeing 100% cell or packet capture and, where loss is unavoidable, knowing not only that packets have been lost but where. The "where" is important because, when analysing a packet trace, it's important to be able to compensate for lost packets when calculating inter-arrival times.
Most commercial NICs keep a count of dropped packets, but they can't indicate where packets were lost. The DAG prepends a header[12] which, amongst other things, indicates how many packets were dropped between that packet and the previously accepted packet.
The DAG is also engineered to deliver recorded packets to the host with the greatest possible efficiency. That, together with the interstitial loss counter, is what makes the DAG so appealing for surveillance applications. The interstitial loss counter also finds application in forensics; a prosecutor needs to be able to prove that the record is complete or, if it is not, where it is not.
Controversy and surveillance
In October 2016, The Intercept published an article showing that Endace customers include intelligence agencies, including the GCHQ, Canadian and Australian intelligence agencies, and the DGST (Morocco's domestic surveillance agency).[13]Edward Snowden documents have shown that the GCHQ has installed massive surveillance of network communications in UK, using the over-sea cable between Europe and North America.
