Investigation of a Novel Data Transfer Methodology via Protein Entanglement Across Mucous Membranes
"What do you know about the information theoretic limits of predictive magic spells?"
Twilight Sparkle turned to her newest Chief Accountant[1] as she asked the question. Starlight looked up from her tablet at Twilight, eyebrows quirked in surprise. "Huh? You're asking me?"
"Well, you're the only person in the room," Twilight said. "And you're also the only expert on Equestrian magic we have here." It almost didn't hurt to say that anymore. "I could use your expertise if you aren't too busy with your work."
"Oh, I finished everything hours ago," Starlight said, poking away at her tablet. "Also, did you know that there are games on this thing? It's a simulation within a simulation! I wonder if one of these games has a game nested inside of it? Then I could be playing a simulation inside a simulation inside a simulation[2]!"
Twilight chose to ignore the second part of Starlight's statement. As long as she did her job and wasn't trying to "test the boundaries of the simulation" again, it wasn't her problem. "In that case, if you're free, I could use another set of eyes on this problem."
"Sure, what's up?"
"I'm getting channel capacity performance on this data transmission system that far exceeds even what the theoretical lossless channel should be achieving, and I think some of these divination enchantments might be the problem."
"Huh, that's weird."
Twilight perked up. "I know, right? Even if magic is some unknown force that violates many of the known laws of physics, it should still be subject to logic and, by extension, mathematics. Have you ever seen something like this in Equestria?"
"Oh, no." Starlight set down the tablet. "I'm just surprised that this world decided to give me a magic problem to solve. Then again, this used to be Twilight's vacation world, so I guess that makes sense. It's a nice change from the action games with the cultists outside."
Despite her earlier commitment to ignoring Starlight's asides, she couldn't in good conscience let that one go. "What action games?" After all, she really didn't want to have to throw Rainbow Dash at the insurance people again.
"The fighting simulator that starts with the cultists trying to grab me," she replied. "It's a great way to burn off all this aggression I have now that I'm one of the good guys."
"That isn't— nevermind." Twilight made a mental note to throw Rainbow Dash at the insurance people again. This would've been so much easier if it were Sunset… no, don't think about that. The past was the past, and Twilight had to look forward, towards the future. "Anyways, I think I built a system that violates one of the fundamental mathematical laws of the universe, and I'm really hoping that isn't the case. So, back to my original question: what do you know about the information theoretic limits of magic?"
"Information… theoretic?" Starlight's head tilted in confusion. "How can information be theoretical?"
"No, I mean like in terms of information entropy and channel capacity and—" Twilight narrowed her eyes at Starlight, who just stared blankly back at her. "Do you understand anything I'm saying or did you just zone out?"
Starlight shook her head. "I have no idea what you're talking about. How can information even have entropy? That's like, a measure of a physical system that you're not supposed to mess with because something something natural order something something abominations against all that is good and pure, right?"
"Information entropy is a separate concept from thermodynamic entropy, though they have many similarities," Twilight said. "Both are related to probability, but information entropy is more of a purely mathematical concept."
"And you think you broke one of the rules about it?" Starlight frowned as Twilight nodded in response to the question. "Okay, that's bad. Magic isn't supposed to be able to break math. Unless..."
"Unless...?" Twilight asked as Starlight trailed off.
"Unless there's something broken in this simulation." Starlight's eyes widened. "Oh no."
"What's wrong?" Twilight had never seen Starlight look this worried before, and it was making her worried too.
"A device that violates the internal consistency of the simulation could destabilize the entire simulation. If that happens, the whole thing could collapse, and who knows what might happen to me if I'm still inside it." Starlight suddenly advanced on Twilight, grabbing her and pulling her close, her wild, panicked eyes staring into hers. "If you die in the simulation, do you die in real life?"
Twilight very carefully extricated herself from Starlight's grasp. "Yes," she said. "If you die here you die in Equestria too." It was probably for the best that she set her straight on that, just in case. "But I don't think it'll come to that. I just need help figuring out how this magic works. There's no danger at all."
"Are you sure?" Starlight asked, her breathing still quickened.
"Very." Returning to the topic at hand, Twilight asked, "So Equestria never developed a mathematical theory for communication? You ponies never figured out the limits of communicating over a radio channel or the limits of data compression technologies?"
"I still have no idea what you're talking about," Starlight said. "Can I pick the newbie option where you explain the rules of this world for people who haven't played before?"
"Of course!" Twilight said, smiling. Even if Starlight had some… odd views about this world, she would hopefully bring back some of these concepts to Equestria. Even if it had nothing to do with beans, Twilight wasn't about to pass up an opportunity to lecture about an entirely new field of study to a willing audience. "So, the field of information theory is rooted in the mathematical analysis of information and its transmission."
"But, information is an abstract concept. I know this simulation needs to be built on rules for it to function, but how do you even measure information using math?" Starlight continued to look skeptical, and Twilight walked over to the whiteboard in her lab as she began her lecture.
"Well, in order to answer that question, we first have to ask a more fundamental one. Namely, what is information, and what does it mean to transmit it?"
"Hmm, I guess transmitting information is like what Spike does when he sends those letters to Celestia." Starlight glanced around the room. "Hey, where is Spike, anyways? I remember he had a counterpart here too, but nobody ever seems to mention him."
"Oh, he's around here somewhere." Twilight gestured vaguely into the air. "I bring him to work most days, but he just fades into the background, and then everyone sort of forgets he exists."
"Huh, this simulation is really accurate."
"Anyways." Twilight cleared her throat to get Starlight's attention and to get back on track. "Seventy years ago, a mathematician trying to determine maximum data transfer rates devised a framework to answer the question of what information was and provide valuable insights into the limits of communications technology[3]. His fundamental insight was to provide an intuitive definition of information that led to useful mathematical results. He came up with his definition by thinking about what it means to transmit information from one person to another."
Twilight picked up a marker, checking that Starlight was still paying attention to her before continuing. "Let's say we want to meet up sometime tomorrow, and we talk about what time the meeting should be today. If we agree to meet each other at noon, then I don't need to tell you anything else tomorrow — you already know the meeting time. We can say that I don't need to transmit anymore information to you tomorrow in order to specify the time.
"On the other hand, if we agree to meet at eight, then there's two possibilities for the meeting time: eight AM or eight PM. In order to know the exact time, I would need to tell you which one of these two times is the meeting time at some point tomorrow. Now, I could do that by sending you a message that says \texttt{8AM} or \texttt{8PM}."
At this point, Twilight wrote out the two pairs of symbols on the board. "But, if I'm trying to save space, then I could just send \texttt{AM} or \texttt{PM}, since you already know that it's at eight." Two more sets of symbols appeared below their counterparts.
"To go even further, we don't really need to send the \texttt{M}, since it doesn't really specify anything. So we just need to send one symbol — \texttt{A} or \texttt{P} — to completely specify our meeting time." As Twilight wrote down the final, minimal messages, she circled them to emphasize their short lengths.
"Now, let's say that we decide to meet some time after noon on the hour, but we don't decide on which hour we're meeting at. This time, there's eleven possible meeting times — one PM, two PM, and so on until eleven PM — and I need to send you at least two symbols to specify the time, since I can't capture all the numbers between one and eleven with just one digit."
This time, Twilight wrote out all the numbers: \texttt{01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11}. She kept the leading zeroes to highlight the fact that the second digit always needed to be communicated.
"If we decide instead that we can meet at any time on the minute after noon tomorrow, then the number of possible meeting times goes up by a factor of sixty, I would need to send you four digits to exactly specify the meeting time.
"So clearly, there's a relationship between the number of possible times that I need to differentiate between, and the length of the message that I send you. A longer message should also, naturally, contain more information. From this, we can say in a general way that information can be defined as something that lets you differentiate between different choices, and that the more information something has, the more things it can differentiate between.
"Likewise, the more choices you have to pick between, the more information you need to specify one of them. An example of a way to measure this would be by looking at the length of the shortest message you need to send in order to perfectly differentiate between all the different available options."
At this point, Starlight raised her hand. "I think I found a bug in the simulation. Or a parameter that wasn't accounted for in its creation. The length of the shortest possible message would depend on what language you're using. Although, I've only heard one language here so far, so maybe it's not a problem."
"No, you make a good point, Starlight." Even if it was insane in its own way. "Yes, depending on how big the set of symbols you can use are, the minimum length of the message needed will be different. Specifically, if your language has a set of possible symbols of size k, then with n symbols you can differentiate between k^n different possibilities. Conversely, if you want to be able to differentiate between M different things, then you need at least \lceil\log_kM\rceil symbols." Once again, Twilight wrote out all the expressions on the board to keep track of them.
"So, for example, if I want to represent 100 different things, and I'm only allowed to send the digits from 0 to 9 — that's ten possible symbols — then I need to transmit \log_{10}100 = 2 symbols to specify which of the 100 things I'm talking about. If I wanted to represent 500 different things, I would obviously need more digits. Specifically, I would need \log_{10}500 \approx 2.699 digits."
Starlight raised her hand again. "Wait, how do you have point-six-nine of a digit? Digits can't be split into pieces, right? Last time I tried that, the other Twilight lectured me about magical ethics for an hour, and I think she mentioned it was impossible, but I couldn't hear her that well over the very rude eldritch screams from The Manifold."
"I'll get to that later, though I'd be interested to hear what you did to try to split a discrete mathematical concept. For now, can you accept that information can be measured in the logarithm of the number of things it can differentiate between?"
"I mean, I guess? But wouldn't any increasing function of the number of possibilities work? Why logarithms? Why not a linear function, or the square root, or a sigmoidal function?"
Twilight rolled her eyes. "As if any information theorist would be caught dead using a sigmoidal function. But yes, you're right, there's other options, but the logarithm is used mostly because of notational and intuitive convenience. A lot of the equations and expressions in information theory end up being closely linked to the logarithm anyways, not to mention that the logarithm has nice properties with respect to addition and multiplication. Intuitively, the logarithm also has a simple interpretation in terms of the number of symbols needed for representation.
"For example, the standard base we use for these logarithms is two, because in computer systems we generally only have access to binary digits, or bits, which can only take on one of two possible values: zero or one. Thus, measuring information in bits lets us directly think about information quantities in a tangible way.
"Now, let's add another complexity. Say I want to send you a number between \texttt{1} and \texttt{4}. How many bits do I need to specify the number?"
Starlight took a moment to think about the answer. "Well, that's four different possibilities you need to differentiate between, and bits use a logarithm with base two, so you'd need \log_{2}4 = 2 bits." Twilight wrote the expression on the board as Starlight spoke.
"Exactly. And if we were allowed to discuss a system for translating bits to numbers, how would you create a rule to translate the two bits I send you into the correct number?"
Another pause. "I could come up with a one-to-one function from bits to numbers. Like, \texttt{00} is \texttt{1}, \texttt{01} is \texttt{2}, \texttt{10} is \texttt{3}, and \texttt{11} is \texttt{4}."
Twilight nodded, writing down the correspondence. "That works. Now, we're going to add two complications that lead to the fundamental measure of self-information. First, instead of sending one number, we're going to send a sequence of them. And you don't get to know how many numbers I'm going to send. Now, how many bits per number do I need to send you?"
"I'm going to say not 2?"
"No, it's still 2 if you have no idea which number I'm going to send at any given time. But — and this is where the metaphorical magic happens — what if we both knew something about the probabilities of the numbers?" Twilight heard her own voice rising with excitement as she neared the grand reveal. "What if I told you that there was a one-half chance that I would send a \texttt{1}, a one-quarter chance that I would send a \texttt{2}, a one-eighth chance that I would send a \texttt{3}, and a one-eighth chance that I would send a \texttt{4}?"
On the board, Twilight wrote out:
P(\texttt{1}) = \frac{1}{2}
P(\texttt{2}) = \frac{1}{4}
P(\texttt{3}) = \frac{1}{8}
P(\texttt{4}) = \frac{1}{8}
"Now, we could use the same system as before: \texttt{00} gets mapped to \texttt{1}, \texttt{01} to \texttt{2}, \texttt{10} to \texttt{3}, and \texttt{11} to \texttt{4}, and we'd need two bits to send each number.
"Or, we could exploit our knowledge of the probabilities. Consider this mapping: \texttt{0} maps to \texttt{1}, \texttt{10} maps to \texttt{2}, \texttt{110} maps to \texttt{3}, and \texttt{111} maps to \texttt{4}." Twilight grinned. "Now, the average number of bits I need to send each number is:"
\texttt{Average number of bits} = P(\texttt{1})\times 1 + P(\texttt{2})\times 2 + P(\texttt{3})\times 3 + P(\texttt{4})\times 3
\texttt{Average number of bits} = \frac{1}{2}\times 1 + \frac{1}{4}\times 2 + \frac{1}{8}\times 3 + \frac{1}{8}\times 3
\texttt{Average number of bits} =1.75
"So you see, by leveraging knowledge of the probabilities, we can get better performance on average compared to a situation where we don't know the probabilities, or where all the probabilities are equal — you can show that if all four numbers had a one-quarter chance of being sent, then you can't do better than 2 bits per number."
"Can't we do even better by saying that \texttt{0} is \texttt{1}, \texttt{1} is \texttt{2}, \texttt{10} is \texttt{3} and \texttt{11} is \texttt{4}?" Starlight interjected.
"Sure, that would lead to fewer bits per number, but consider this: say I send you a sequence that says \texttt{1010}. Now, did I send you \texttt{33}, \texttt{2121}, \texttt{213}, or \texttt{321}?"
"Oh." Starlight's face twisted up in thought. "Huh, I guess you can't."
"Yeah, you need to be careful when designing a mapping for sequences, since you need to make sure that there's no ambiguity. Anyways, now that we can show the benefits of integrating probability into a communications system, we need to extend our definition of information to include this.
"Our updated measure of information would have to have several properties:
"Say we have a set of n possible outcomes we want to differentiate between, and we have probabilities \{p_1,p_2,...p_n\} for all n outcomes. First, our measure needs to be continuous with respect to each p_i. It wouldn't make sense for an infinitesimal change in probabilities to result in a sudden jump in information. Second, if all the probabilities are the same — that is, p_i = \frac{1}{n} for all i=1,...,n — then the measure of information should be monotonically increasing with respect to n, just like you said before.
"Finally, we want the measure to be such that if we can decompose the outcomes into two stages, then we should be able to write the measure as a weighted sum of the information measures in each stage. For example, consider these two probability distributions:
"You can see that the probabilities of the final outcomes are the same in both cases. Specifically, p_a = \frac{1}{2},p_b = \frac{1}{3},p_c = \frac{1}{6}. However, in the second example, we decomposed the probabilities into two stages. In the first stage, we basically flip a coin to generate a fifty-fifty outcome between a and d. Then, if we get, d, we will generate another coin flip, this time a biased one, to generate the final outcome.
"Since the final outcomes are the same, we should have that:"
Info( \frac{1}{2},\frac{1}{3},\frac{1}{6}) = Info( \frac{1}{2},\frac{1}{2}) + \frac{1}{2}Info( \frac{1}{3},\frac{1}{6})
"Where the \frac{1}{2} weighting on the second term on the right-hand side comes from the fact that the second coin flip only happens half of the time. Are you with me so far?"
Starlight shrugged. "Sure. Continuity, monotonicity, and some kind of additivity. Throw in completeness, a null set, and the Triangle Equality and you've got all the elements of a good measure."
"Umm, don't you mean the Triangle Inequality?"
"Oh, right, yeah." Starlight stared awkwardly down at her feet, her next words muttered under her breath. "Stupid fabric of reality messing with Equality Maths."
"But yes, continuity, monotonicity, and additivity are the three main properties we're looking for, and it turns out that there's only one mathematical function that obeys all three constraints. We call it entropy, and denote it with the symbol H. H is defined as follows:"
H = -\sum_{i=1}^np_i\log p_i
"Normally, we apply H to a random variable, which we can denote as X, to get:"
H(X) = -\sum_{x \in X}p(x)\log p(x)
"Where the sum is over all possible values that X can take."
Starlight perked up as she studied the expressions. "Oh, I see! It's just like thermodynamic entropy!"
"Exactly." Twilight beamed at how fast her pupil was picking this up. "And it has some really nice properties. First of all, H(X) is equal to zero if and only if X is deterministic and can only take on one possible value. Otherwise, H(X) is positive. This is good for optimization purposes.
"Secondly, H is maximized for a given n if all outcomes are equiprobable. That is, if p_i = \frac{1}{n} for all i. This makes sense, since when all outcomes have the same probability, you can't leverage the probabilities at all, and so you need the maximum amount of information to differentiate between outcomes.
"There's some other nice properties, like how the joint entropy H(X,Y)=-\sum_{y \in Y}\sum_{x \in X}p(x,y)\log p(x,y) is always less than or equal to H(X) + H(Y), or how conditioning always reduces entropy[4]." She wrote the two properties on the board.
H(X,Y) \leq H(X) + H(Y)
H(X|Y) = -\sum_{x \in X}\sum_{y \in Y}p(x,y)\log p(x|y) \leq H(X)
"Both of these basically imply that you can't lose information by considering some sort of side information. Like, say X is the quantity of beans sold in Canterlot last year. This information has a certain amount of entropy. If I also know, say, the number of marital aids sold last year, and I call that Y, then I can use that information to get a better idea of what bean sales would have been like, and thus I get less additional information from also knowing X than I would have if I knew nothing from the start. Even if I let Y be something completely irrelevant like average temperature in Fiji, I won't somehow get more information from X by knowing Y."
For a moment, Twilight braced herself for some offhand comment about how she somehow managed to bring beans into everything, no matter how unrelated. Then she realized who she was talking to, and her heart felt heavy once again as she continued her lecture.
"Finally, if we use logarithms with base 2, then the entropy of a source of information is equal to the minimum number of bits per symbol we need to communicate a sequence of infinite length from that source through a lossless channel. Or, in other words, it's the minimum amount of bits we can compress the source into for storage[3]."
That got Starlight's attention. "Wait, you're telling me that this one quantity is a hard limit on how much you can compress information? Just like that? No wonder this simulation is unstable if it's built on rules that're that simple."
Twilight smirked, though her heart wasn't really in it. "I can show you the full proof if you want later, but the outline of it is as follows:
"Consider the problem where I want to send N symbols from some probabilistic source with n possible symbols and symbol probabilities p_1, p_2, ..., p_n. Now, are you familiar with the Law of Large Numbers?"
Starlight rolled her eyes. "Duh. I'm a magical prodigy — of course I know what the Law of Large Numbers is."
"Okay, just checking. So, in that case, you know that as N becomes large, the Law of Large Numbers dictates that the number of occurrences of the first symbol in the sequence will be close to p_1N with near certainty, and the number of occurrences of the second symbol will be close to p_2N with near certainty, and so on.
"Thus, the probability of a typical sequence occurring is:"
p = p_{1}^{p_1N}p_{2}^{p_2N}...p_{n}^{p_nN}
"Taking logs, we get:"
\log(p) = N \sum_{i=1}^{n}p_{i}\log(p_i)
\log(p) = -NH
H = \frac{\log(1/p)}{N}
"So, since the number of typical sequences is around 1/p, and the probability of a non-typical sequence goes to zero as N approaches infinity, we can see that the entropy is equal to the average number of bits needed per symbol to transmit the sequence under ideal circumstances."
"That's incredible." Starlight stared at the board with wide eyes. "When I get back to Equestria I need to tell Twilight that her simulation invented a new field of math! This is going to be huge."
"Glad to help," Twilight said. "If you want, I can lend you some books that go more in-depth into the subject."
"Yeah, sure." Starlight said absently as she continued to stare at the board. "I'm gonna need to memorize all of this information to bring back."
"I think the portal lets you bring back objects, so you can just bring the book through.”
"Oh, that makes things easy." Starlight returned her attention to Twilight. "So, what problem do you have?"
"Right." In her excitement, Twilight had completely forgotten about the original reason why she'd begun talking to Starlight. "I'm getting performance that breaks the entropy limit for a data transmission channel I'm working on. There's a magical component to the system, which is why I wanted your expertise."
"Okay, what's your system like?"
"Well, I have a spell that converts a bit stream into a sequence of proteins, a mechanical system to transfer the proteins to a decoding component, and a spell at the decoding module to convert the proteins back into a bitstream via a magical bio-electric interface."
"And you're transmitting more bits than you're supposed to be able to?"
"That's the thing," Twilight said. "It doesn't always happen. If I use a random number generator to generate a stream of bits with equal probabilities of being zero and one, then the channel performs as expected. Because some of the proteins degrade, some of the information gets lost, so I need to put in extra coding bits — extra proteins, in this case — to be able to correct any errors. So the number of bits per protein I get out is not exactly the logarithm of the number of possible proteins in a chain.
"But, if I try to use it to transmit actual data from a real-life source, then suddenly the number of bits in the output stream per protein is more than the logarithm of the number of possible configurations."
A thoughtful look crossed Starlight's face, and Twilight waited patiently for her to process the explanation. "You said that a bit stream can be compressed to entropy, right?" she finally asked, and Twilight nodded. "If the spell is Divination-based, it can search over all possible configurations to be able to achieve this entropy limit. Or however close you can get to it without going to infinity."
"But the bitstream is already compressed," Twilight said. "I took the original data and used a universal compression algorithm to output a sequence with independent and equiprobable bits, so in both the real data case and the synthetic data case, the input to the system is a sequence of bits that cannot be entropy-coded further."
"Huh, if your Divination spell is only connected to the input and output streams, then I can't see how it can bypass a physical limitation like that." Starlight shook her head. "Of course, the spell could be interacting with the simulation itself. If you tore a rift into reality you could take a peek and see for yourself."
"No!" Twilight winced at the loudness of her own voice before forcing herself to quiet down. "No more rifts in reality. Can you just assume that this isn't a simulation, please?"
"Fine." Starlight sighed. "I guess it's not fun if I cheat anyways, and I don't know if there's a bodycount penalty in this challenge. Okay, can you describe the actual physical system then? Maybe there's a parameter you're not seeing."
"Of course." Keep it together, Twilight. You just need her expertise to solve this problem and then you can go back to being alone. Again. "It's actually related to a new product I'm hoping to launch by the end of the next quarter. I was investigating the properties of the Datanis, and I discovered that it had the potential to act as an incredibly effective data transmission device — though not to a mathematically impossible extent — but without a method to translate the payload of the device to something computer-readable, there are few practical applications."
"The Datanis," Starlight said, "that's the Beanis that jizzes information, right[5]? I should've known that's what you were talking about when you mentioned protein sequences."
Twilight nodded. "Yes, that's the working name for our data transmission Beanis products. Anyways, in order to realize the full potential of the Datanis, I created an organic datastream extractor and decoder."
She opened her desk drawer and pulled out a long metal cylinder with wires protruding from one end that ended in a USB plug. "This is the Translegume Masturbatory Infracoder, or TMI for short," she said, turning it so that the other end of the device faced Starlight. A small slit had been cut into the otherwise smooth brown substrate on that end. "It contains a port that's lined with magically-enhanced enzymes that break down and analyze data-encoded protein sequences. The primary structural component of the port is composed of a blend of several hybrid species of beans bred for optimized thaumo-electrical conductivity."
Starlight studied the TMI for several seconds. "So you invented an organic fleshlight that also doubles as a protein sequencer and, I'm assuming, a binary decoder?"
"Exactly," Twilight said. "The Datanis is magically inert when it transfers its payload into the TMI, so there shouldn't be any spell interactions between the encoder and the decoder. And yet, I still end up with too much information."
"Huh." Starlight suddenly perked up. "Wait, if the information is stored in the Datanis' semen, then you need to make it cum into the TMI to make the transfer, right?"
Twilight blushed slightly. "Yes," she replied stiffly, "the Datanis must be stimulated into ejaculating its payload inside the TMI."
"And you tested this yourself?"
"Manually," Twilight confirmed. "Why?"
"I think I might have figured out the problem," Starlight replied. "When you said you used real-world data to test the system, what exactly did you compress into the bitstream?"
"Well, in order to push the channel to its limit, I needed to generate a large number of bits. Video data, even when compressed, still has a very high data rate, especially with high-resolution video. I used a database of videos that I had on hand in order to generate the bitstream."
"What kind of videos?"
"Umm..." Twilight trailed off, her voice almost becoming a whisper as she answered Starlight. "Adult videos."
"You tried to transmit a bunch of porn videos, then?" There was no judgement or surprise in Starlight's tone, just curiosity.
Twilight nodded, unable to voice her answer.
"And you watched all the videos already, right?"
"I-I needed to perform research on p-possible use cases," Twilight stammered. "I discovered at least two new orifices that our research team had previously not considered inserting a Beanis product into."
"Wow, TMI. I mean, too much information, not your thing, and — oh, I get it! But yeah, that explains it." Starlight pointed at Twilight. "The Datanis might be magically inert, but you aren't. Divination spells tend to reach out to living sources of magic to augment their predictions, and since you're holding the TMI and the Datanis, it can probably do some extra prediction that you aren't accounting for."
Twilight's eyes widened. "Conditioning reduces entropy!" She rushed back to the whiteboard. "Of course! If the spell has a connection to me, then it has extra side information that can be exploited. Sleeping and Wolf proved that in the lossless case, if a decoder has knowledge of a source Y, then the upper bound on the number of bits needed to encode a datastream X is given by H(X|Y)[6][7]. This should be the baseline I compare performance to, not H(X)."
The new expressions were written onto the board. Twilight's mind raced as she took in this new development. "Obviously, estimating H(X|Y) is impossible, since it's bound to my memories. I need to treat this like a psychology study, with double-blind tests."
"Or, you can use someone magically inert to test it, like me," Starlight said. "The simulation got rid of all my magic. I guess it's so that I could relax without tearing this world apart to figure out how it ticks? Anyways, do you have a system ready to test out?"
"I do." Twilight plugged in the TMI, turned on the testing program, and then withdrew a pre-loaded Datanis from her drawer. "Do you want to do it now?"
Starlight shrugged. "Sure." She grabbed the Datanis and the TMI from Twilight's hand, and without further ado, began enthusiastically working both devices with a level of dexterity that really made Twilight question what her normally-a-unicorn Chief Accountant did when not working in her office. "Man, this is a lot of fun. I guess that means the problem's solved and this is my reward, right? We saved the simulation?"
As her screen filled up with data showing a much more reasonable transfer rate, Twilight let out a soft sigh. "Yeah, sure." It seemed that Starlight was right, and as a result, Beanis Inc. would be moving on schedule for the development of the TMI. "Congratulations on beating the challenge." Everything had gone smoothly — even if Starlight hadn't been familiar with information theory, she'd picked up the ideas quickly enough to solve Twilight's problem, and was even helping Twilight test the solution. It was the height of efficient operation.
And yet…
Twilight shook her head. There wasn't any reason to think about what-ifs. Sunset was gone, and there wasn't even anything more she could've done. All Twilight was missing out on was Sunset's exasperation at a math problem turning inevitably into something phallic and bean-related. She didn't need her cute, annoyed frown or the sound of her tired, sultry voice or her dedication to making this company succeed despite her revulsion for the products they produced.
It was better this way.
As she began to calculate the capacity of the noisy channel model from the data transfer results, Twilight almost believed those words.
Works Cited:
[1] F. Furvert, "Couch Gagging", fimfiction.net, June 25, 2018. [Online]. Available: https://www.fimfiction.net/story/412255/couch-gagging. [Accessed July 15, 2018].
[2] M. Syeekoh, "Ethical Quandaries", fimfiction.net, Oct. 3, 2017. [Online]. Available: https://www.fimfiction.net/story/387485/ethical-quandaries. [Accessed July 15, 2018].
[3] C. E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., 27:379–423,623–656, 1948.
[4] T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
[5] M. Syeekoh, "Homework is a Real Kick in the Dick", fimfiction.net, Nov 18, 2017. [Online]. Available: https://www.fimfiction.net/story/392337/homework-is-a-real-kick-in-the-dick. [Accessed July 15, 2018].
[6] D. Slepian and J. K. Wolf. Noiseless coding of correlated information sources. IEEE Trans. Inf. Theory, IT-19:471–480, 1973.
[7] A. Wyner. On source coding with side information at the decoder. IEEE Trans. Inf. Theory, IT-21:294–300, 1975.
Author's Note
Follow-up blog here.
Information Theory is a fascinating field, and one close to my heart. For those of you interested in further exploring how we mathematically quantify the ways we represent, transmit, and store information, Thomas and Cover's Elements of Information Theory is a comprehensive yet accessible textbook that can easily be found online.
For those looking for something a little shorter, this document is a good (albeit somewhat dated) primer on the core ideas of Info Theory and some basic applications: http://www-public.imtbs-tsp.eu/~uro/cours-pdf/poly.pdf.
Shannon's original paper "A mathematical theory of communication" is also quite accessible and short, though his notation and terminology are a bit odd by modern standards. I'd recommend the paper to those who already have some familiarity with the field.
EDIT: Thanks to R5h for linking this wonderful xkcd what-if to me. It explains the fundamental concepts behind bits and representing information much more elegantly than I have, with an intuitive link back to the English language.
EDIT2: For the more mathematically inclined, these slides from a lecture at Stanford are nice, compact introduction to the topic as well. A basic understanding of probability theory and random variables/random processes is all you need. These slides also cover an intro to channel coding and stuff like the binary symmetric/erasure channels.
