How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds 129
colinneagle (2544914) writes "What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? In a Network World post, blogger Steve Patterson breaks down a recent breakthrough in stateless transmission using Random Linear Network Coding, or RLNC, which led to a joint venture between researchers at MIT, Caltech, and the University of Aalborg in Denmark called Code On Technologies.
The RLNC-encoded transmission improved video quality because packet loss in the RLNC case did not require the retransmission of lost packets. The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.
In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet."
The RLNC-encoded transmission improved video quality because packet loss in the RLNC case did not require the retransmission of lost packets. The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.
In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet."
shifts the reliance (Score:1)
and what if the coefficients get lost or corrupted?
Re: shifts the reliance (Score:1)
It's either the entire packet comes or it's lost.
Re: (Score:2)
Acknowledgment is conceptually a separate thing from retransmission requests. If the normal transmission stream is such that dropped packets can be detected, then the only return communication necessary is the occasional "please repeat packet X" Of course you probably still want *some* acknowledgment to avoid the scenario where someone watches only the first 5 seconds of video before shutting off their computer but the server keeps streaming video for the next hour to a now non-existent client, but it nee
Re: (Score:2)
I wondered why a system would use it...if I understand it correctly...
it's a new encoding that requires software on either end for TCP, that's in TFA...
Can we update the title please? (Score:4, Interesting)
"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.
And why the "oooh - flappy birds on my phone might be faster" slant? I want a faster SAN!.
Re:Can we update the title please? (Score:4, Insightful)
Re: (Score:2)
I saw this and was looking for the tech details. It looks like they are putting in FEC at the application layer. It's been done before. They are just hiding it better to not have to explain why 1/8th of the data is padding (or whatever ratio they are using).
Re: (Score:1)
Probably from a misleading name. For example, Golay codes are perfect codes. Despite being perfect, there are better choices.
Re: (Score:2)
Re:Can we update the title please? (Score:4, Informative)
Because R-S assumes uniformly random error distribution which is usually not the case when it comes to wireless interference.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
>"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.
Well, yeah. There's lots of things better than Reed-Solomon for doing video encoding in a lossy environment. I worked on such a project at UCSD in 1998 or so with John Rogers and Paul Chu that could eat pretty impressive amounts of noise and still have a usable video signal without retransmissions. It's start getting pretty muddy looking when you really cranked up the losses, but unlike JPEG o
Nothing new under the sun, just new uses (Score:4, Insightful)
Sounds like Parchive from usenet, which is a really good idea in a lossy environment now that I think about it.
Re:Nothing new under the sun, just new uses (Score:5, Informative)
Re:Nothing new under the sun, just new uses (Score:5, Informative)
And like par2, it's going to require a healthy amount of processing from your CPU
The trends to higher-performance multicore processors and parallel operations everywhere in the network and on mobile devices lends itself to an encoding scheme utilizing linear algebra and matrix equations that might not have been possible in the past.
Notice they talk about multicore processors and not some hardware decoding embedded in the networking chip.
From their published paper
Abstract-- Random Linear Network Coding (RLNC) provides
a theoretically efficient method for coding. The drawbacks associ-
ated with it are the complexity of the decoding and the overhead
resulting from the encoding vector. Increasing the field size and
generation size presents a fundamental trade-off between packet-
based throughput and operational overhead. On the one hand,
decreasing the probability of transmitting redundant packets is
beneficial for throughput and, consequently, reduces transmission
energy. On the other hand, the decoding complexity and amount
of header overhead increase with field size and generation
length, leading to higher energy consumption. Therefore, the
optimal trade-off is system and topology dependent, as it depends
on the cost in energy of performing coding operations versus
transmitting data. We show that moderate field sizes are the
correct choice when trade-offs are considered. The results show
that sparse binary codes perform the best, unless the generation
size is very low.
Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.
Re: (Score:2)
Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.
I don't see this as a problem at all. This is the type of thing that screams for a dedicated processor, not a GP chip, and it's typical to see a 99% reduction in power consumption when you go this route.
Re: (Score:2)
Re: (Score:2)
Indeed. Back in the day my company used custom hardware - 8x8 binary convolver - to do pattern recognition on images. It ran at the unbelievable rate of 12 billion pixel operations per second. The chips were originally from the cruise missile program, so we had to get a DoD license for every one we shipped. By the mid-1990s a reasonably fast Pentium could do the same work in software. And I'm pretty sure that some ray tracing code that ran on the Cray X-MP would now run faster on my phone.
Re: (Score:2)
what the FEC... (Score:4, Funny)
Re: (Score:3)
FEC generally does not seek to recover lost data, only the proper state of flipped bits.
Re: (Score:2)
There's no difference between the two (what's the difference between a 1 flipped to a 0, or a 1 that was lost and so interpreted as a 0?), and FEC is frequently (typically?) used to recover lost data. Interleaving is often used such that losing an entire transmitted packet results only in losing small parts of multiple actual packets, which FEC can then compensate for.
Re: (Score:2)
FEC algorithms do not treat (an unknown number) of lost bits as a stream of 0 bits, since it can't know how many bits are lost.
Re: (Score:2)
They don't need to. The underlying layer knows how many bits are missing based on the timing.
Re: (Score:2)
Re: (Score:3)
What people are saying is they have implemented FEC at the packet level. I don't know much about wireless but I do remember the maths lectures I attended 25yrs ago on error correction techniques. At the time I could perform the FEC algorithm by hand, I consider myself "mathematically inclined" and have been awarded bits of paper attesting to that inclination but to this day I still have just enough understanding of the mathemat
Re: (Score:2)
I would say FEC to is correct the current bit state, while this method RLNC is to correct a past or future packet.. I am totally suggesting is with fast enough processing power the protocol would assume every packet is missing and have recovered packets "ready to go" if one is missing...
Re: (Score:2)
Right. (Score:5, Insightful)
This is yet another form of forward error correction. The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.
This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.
Re: (Score:3)
The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.
Yeah, that would be impossible. But I don't think they meant to claim that there is no extra data. When they say "The combined packet length is no longer than either of the two packets from which it is composed." I took that to mean that each packet is a fixed length. Not that they didn't add extra data to get to that packet length.
This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.
Yeah. It might be good for for wireless networks, where even a "5 bar" wireless network connection has a fairly consistent 1% packet loss. The QUIC [connectify.me] protocol is another atte
Re: (Score:2)
What if you could transmit data without link layer flow control bogging down throughput with retransmission requests
TFS makes it look like network coding can magically send data without any form of ACK & Retransmit. Network coding still requires feedback from a flow control protocol. You need to tweak the percentage of extra packets to send based on the number of useful / useless packets arriving, and you still need to control the overall rate of transmitted packets to avoid congestion. The goal is to make sure that every packet that arrives is useful for decoding the stream, regardless of which packets are lost. So
Re: (Score:2)
Re: (Score:2)
I'm thinking of Network Coding Meets TCP [google.com]. Though that doesn't give a great background. I've experimented with my own implementation, but had to shelve it due to lack of time. I'll try to quickly summarise the core idea;
You have packets [p1, .. pn] in your stream's transmission window.
Randomly pick coefficients [x1, ..., xn] and calculate a new packet = x1*p1 + x2*p2 + , .... (in a galois number field, so '+' is an XOR and '*' is a pre-computed lookup table). Sending the coefficients in the packet header.
Re: (Score:2)
they've invented forward error correction, this could enable data communications and audio CDs.
I think they combined FEC with TCP.
TCP sends packet and then waits for ACK. However they eliminated the ACK part and just added extra parity in the packets. So, even when packet is lost, it can be reconstructed.
Of course there is a probability that multiple packets are lost so that it cannot be reconstructed. Of course, there is more redundant data in the stream instead of the small ACK packets. The packet loss characteristics of the channel probably makes this approach more efficient.
Re: (Score:2)
Re: (Score:1)
So this is basically an efficient redundancy transmissiin scheme that out-performs (in spite of using extra data) retransmission.
At a 3% artificially-induced error rate. What about real-world conditions? Wouldn't this clog a network more, not less?
in simplified terms, it's forward error correction (Score:2)
And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?
This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?
Re: (Score:2)
yeah TFA says it's link-layer flow control:
it can "piggyback" on TC-IP but you have to use their "software" on either end
Re: (Score:3)
Many of the details fail under closer examination. It doesn't "use" TCP-IP. It could use a propritary IP stack that is TCP/IP compatible. So it'll use IP addresses and port numbers like a TCP packet would so that switches and routers in the middle wouldn't know or care what it's doing. If they put it on an IPX/SPX core it'd fail to route across the Internet. But it isn't TCP. It doesn't use a TCP compliant stack. It just looks like one to the outside world. It will not re-t
Comment removed (Score:4, Informative)
Re: (Score:2)
And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.
How about "UDP is a member of the TCP/IP suite of protocols"? Though most leave off the "suite of protocols" at the end, it's implied. I know lots of people smarter than you that would say "UDP is a subset of TCP/IP" or some other wording to that effect. Even Wikipedia [wikipedia.org] agrees with me. TCP/IP is shorthand for The Internet Protocol Suite, which includes UDP. So UDP "runs over" TCP/IP
Re: (Score:2)
I think the writer is confused. This sounds like a non-routed layer 2 error-correction protocol for error prone networks, like cell phone data and Wi-Fi. The only relation this has to the Internet is that it can carry TCP/IP traffic, just like Ethernet, Frame Relay, ATM, and any number of other layer 2 protocols can carry TCP/IP.
Re: (Score:2)
Re: (Score:2)
Yeah, they probably are using UDP in some video tests, but I think that is irrelevant. If this is a competing technology to Reed-Solomon encoding, then it is almost certainly a data-link layer protocol and is agnostic to network or session protocols.
From Wikipedia:
Reed–Solomon codes have since found important applications from deep-space communication to consumer electronics. They are prominently used in consumer electronics such as CDs, DVDs, Blu-ray Discs, in data transmission technologies such as DSL and WiMAX, in broadcast systems such as DVB and ATSC, and in computer applications such as RAID 6 systems.
I think the writer's mistake was in seeing everything through Internet-colored glasses. The article specifically says "link layer", not "session layer". TCP is a session layer technology. My best guess from the article is that this is not an e
Re: (Score:2)
It is an error correction protocol to reduce OSI layer 2 packet (or more properly, "frame") retransmission.
But the "frame" doesn't exist end-to-end, but is re-created for each network segment. So their statements that it runs on the endpoints and doesn't require any changes in the middle indicates it is a higher layer protocol. Some of the layer 2 already include FEC. So why have double FEC? (it's bad). It's not double FEC at the same layer, but an application-layer FEC (based on their description). You can have them on different layers because they are correcting different things. L-2 FEC is correcting bit
Re: (Score:2)
And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?
This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?
I believe the issue is that you can't sell it to the cable companies and the DSL providers that implement PPPOE in order to track your surfing, to make sure you are buying your television programming from them, rather than file sharing, and they can intentionally make things like Tor not work. Not that PMTUD works on those things unless the modem proxies the ICMP messages, which are usually blocked by the cable companies, unless you explicitly ifconfig down to 1492 yourself, or enable active probing for bl
sounds like an ad for the future fast lane (Score:5, Funny)
Re: (Score:2)
Engineers want to try it because it's a cool idea. Biz, focused on profit, doesn't. This is why AT&T rejected Kleinrock and others who came to them with ideas of the internet, The market delayed implementation out of short-sighted, blinkered concentration on profit as opposed to the General Welfare.
Re: (Score:2)
With net neutrality, the incentive for trying such ideas will be nonexistent.
As long as there is a significant bottleneck somewhere in the last mile or so (typically 4G/LTE or crowded WiFi) the incentive for efficient use of the available bandwidth will remain huge, regardless of politics.
The thing that might hamper development is if one company manages to get a monopoly on the creative content so that everyone have to use their service regardless of how shitty it performs.
Re: (Score:2)
GAAA! (Score:1)
In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.
If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.
Re: (Score:2)
Re: (Score:2)
If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.
If it is like most such articles, they use 35 pages of polynomial math over a GF(2) field as a way to mathematically formalize what could be better explained with a one-page circuit diagram with a few XOR and shift registers and one paragraph of commentary, so yeah, wait for the boiled down version.
Re: (Score:2)
There's no single paper. There's a lot of papers that refine the technique and apply it to different problems. To even begin to understand it, you must have a good grasp of network coding as it's actually applied in real life. The butterfly network is but a starting point.
Word (Score:5, Funny)
"the immediately earlier sequenced packet". There a word for that. It's called "previous". As in "the previous packet".
Re: (Score:1)
Head east, Dan. Head! East!
Nothing says the previous packet has the previous sequence number. Packets may arrive in any order, or not arrive at all (the point here). The stack makes it all work at the ends. This is all layman should need to understand, and as you have shown, all they can be expected to understand. It was so wored for a reason, and it it's lost on the layman, that's all right.
Re: (Score:2)
Nothing says the previous packet has the previous sequence number
The word "previous" says the previous packet has the previous sequence number. I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.
However I would say is that nothing says that the "immediately earlier" packet has the previous sequence number.
We could get pedantic and talk about sequentially previous and chronologically previous, further clarifying that chronologically previous is from the point of view of the receiver (since
Re: (Score:1)
Sure, just like readers are able to reconstruct the missing second word in your post. But the word "previous" has some ambiguity. The original wording is more precise and specific.
Re: (Score:2)
But I agree in principle, the whole thing was poorly worded. Like it was written by a writer that doesn't understand it. That was dictated to by an engineer who was ordered to not be specific about it.
Re: (Score:2)
When you're talking about packet protocols, where things get lost, duplicated, or reordered, "previous" is an incredibly imprecise word.
Re: (Score:3)
...so does "earlier"
Re: (Score:1)
Thus the additional qualifiers "immediately" and "sequenced".
Re: (Score:2)
Network Coding (Score:2)
Good Improvement (Score:1)
More research needed (Score:1)
How MIT and Caltech's Coding Breakthrough Could Accelerate Mobile Network Speeds
I wish they also made some breakthrough to increase the data caps we are stuck with.
Re: (Score:2)
caps are Layer 8 problem
random? (Score:1)
Re: (Score:2)
Packet loss models? (Score:3)
If you assume that every packet has a k/n chance of being lost, then being able to reconstruct a single missing packet could be incredibly useful. However cell phone packet losses tend to be incredibly bursty, i.e. they will have a very throughput for a while, but then all of a sudden(maybe you changed towers or went under a bridge etc) lose a whole ton of packets. Can this algorithm deal with bursty losses? I wish TFA was a bit more clear on that
Re: (Score:2)
(I haven't read this paper yet, but I've read other Network Coding data and experimented with the idea myself)
With TCP, when a packet is lost you have to retransmit it. You could simply duplicate all packets, like using RAID1 for HDD redundancy. But this obviously wastes bandwidth.
Network coding combines packets in a more intelligent way. More like RAID5 / RAID6 with continuous adaptation based on network conditions. Any packet that arrives may allow you to deduce information that you haven't seen before.
A better explanation (Score:2)
The story linked to seems to have an awful explanation of what's going on. This makes a lot more sense:
http://www.codeontechnologies.... [codeontechnologies.com]
Reminds me a little of a random project I started back in college where I'd transmit a file in a bunch of packets where each contained the original file modulo a specified prime number. That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of th
Re: (Score:2)
It might be possible now with IPv6.
Re: (Score:2)
Guess what: According to the Wikipedia article about TCP, that's what the "selective acknowledgment" (SACK) option defined in RFC 2018 does. So _poof_ goes the benefit of this sc
Really? (Score:2)
"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file."
So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file? Thats impressive. Which college was this, Hogworts?
Re: (Score:2)
So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file?
Considering each one is only sent once, that would be some feirce level of broken compound load balancing.
Note he said each packet contains the modulus of the entire original file with a different prime. The only thing that would cause duplicate packets would be running out of acceptably-sized primes.
Re: (Score:2)
You need to re-read what he wrote:
"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them "
How can you split something into 10K packets then get *any* 10K of them? He's either saying he sends more than 10K or the receiver has to receive all 10K packets that were sent to reconstruct the file which I doubt is what he meant.
Re: Really? (Score:2)
Yeah that could have been more clear. I'm suggesting sending a constant stream of packets until the target asks you to stop. No need for windowing or flow control, but pretty computationally expensive
Excellent! (Score:2)
This means my mobile internet speed might soon be up to 10 bps instead of the 2 bps I seem to get at the moment!
Re: (Score:1)
It means that your ISP can oversell their bandwidth even more.
I've seen this before (Score:2)
Except it's called MPEG video. And it's used for TV.
MPEG also has a mode to recover the errors, but it's expensive, and when you're streaming, who cares? If your link sucks, you don't blame the stream.
Good press release / slashvertisement (Score:1)
"Code On" has done well at generating some buzz.
Unfortunately, the only details on their website are of the type "this is awesome", with no description of how the breakthrough works.
http://www.codeontechnologies.com/technology/white-papers/
I just wish they would post actual details on the encoding method. How does it compare to Fountain Codes such as RFC 5053?
That's the over-simplified version? (Score:2)
In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.
Uh... could you simplify it just a little more?
How does a "later-sequenced packet [...] include earlier-sequenced packets"?
Re: (Score:2)
They say "randomly" generated coefficients, but I'll bet you can use a psudo random number generator and pass in the same seed value to both the sender and reciever. Bam, now both sides have the same set of semi random coefficients to use when doing the fancy linear algebra.
Re: (Score:2)
The linear algebra isn't even fancy. They need to do a lot of linear equation solving, that's about it.
Time saver (Score:1)
parses like a teaspoon of sugar (Score:3)
I've been parsing this kind of press release for a long, long time now. I can pretty much tell what we're dealing with by how hard it is to state the advantage of a new approach in narrow and precise language.
That this blurb doesn't even disclose the error model class (error correction is undefined without doing so) suggests that the main advantage of this codec lies more in the targeting of a particular loss model than a general advance in mathematical power.
Any error correction method looks good when fed data that exactly corresponds to the loss model over which it directly optimises.
The innovators of this might have something valuable, but they are clearly trying to construe it as more than it is. This suggests that there are other, equally viable ways to skin this particular cat.
Sounds like a Fountain Code to me (Score:2)
This sounds exactly like a Fountain Code [wikipedia.org] to me, which isn't exactly news.
Re: (Score:2)
yes, by sending packet twice the size
Re: (Score:2)
Re: (Score:2)
Why would you put that on top of a streaming protocol like TCP/IP that's got all the flow control and retry logic happening under the covers? Surely you'd run RLNC over UDP!
UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.
Re: (Score:1)
UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.
UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.
Re: (Score:2)
UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.
http://en.wikipedia.org/wiki/I... [wikipedia.org] TCP/IP is shorthand for The Internet Protocol Suite. UDP is a protocol in The Internet Protocol Suite. Therefore, you are 100% wrong, in your wording, UDP is a subset of TCP. You just chose to leave off the "/IP" because you know you are wrong.
Re: (Score:2)
Usually in such schemes you can recover a packet if you have the surrounding packets before and after it.
Re: (Score:1)