Isolating Sources of Disentanglement in Variational Autoencoders. Hierarchical Long-term Video Prediction without Supervision. Neural Style Transfer via Meta Networks. Frame-Recurrent Video Super-Resolution. Decorrelated Batch Normalization. Discriminability Objective for Training Descriptive Captions. Learning towards Minimum Hyperspherical Energy. Wasserstein Introspective Neural Networks. Self-produced Guidance for Weakly-supervised Object Localization. Measuring abstract reasoning in neural networks.
Generative Adversarial Networks with Python
Efficient end-to-end learning for quantizable representations. Surface Networks. Learning-based Video Motion Magnification. Neural Autoregressive Flows. Weakly- and Semi-Supervised Panoptic Segmentation. Video Re-localization. Real-time 'Actor-Critic' Tracking. Visualizing and Understanding Atari Agents. Image Manipulation with Perceptual Discriminators. Zero-Shot Object Detection. Partial Adversarial Domain Adaptation. Learning to Blend Photos. Generalisation in humans and deep neural networks.
Interpretable Convolutional Neural Networks. Generative Adversarial Perturbations. The Sound of Pixels.
Mutual Information Neural Estimation. Learning to Evaluate Image Captioning. Hyperbolic Neural Networks. Disentangling by Factorising. Tangent Convolutions for Dense Prediction in 3D. Graphical Generative Adversarial Networks. Deep One-Class Classification. Deep Regression Tracking with Shrinkage Loss. Adversarial Logit Pairing. Deep Randomized Ensembles for Metric Learning. Quaternion Convolutional Neural Networks. Single Shot Scene Text Retrieval. Who Let the Dogs Out? Interpretable Intuitive Physics Model.
Between-Class Learning for Image Classification. Conditional Image-to-Image Translation. First Order Generative Adversarial Networks.
Anonymous Walk Embeddings. Learning to Multitask. Learning to Promote Saliency Detectors. Dimensionality-Driven Learning with Noisy Labels. Objects that Sound. Conditional Image-Text Embedding Networks. Bayesian Optimization of Combinatorial Structures. Revisiting Deep Intrinsic Image Decompositions. Adversarial Attack on Graph Structured Data.
Coded Sparse Matrix Multiplication. Hashing as Tie-Aware Learning to Rank. Pose Proposal Networks. Open Set Domain Adaptation by Backpropagation. Neural Sign Language Translation. Efficient Neural Audio Synthesis. Image Transformer. Learning to Understand Image Blur. Learning and Using the Arrow of Time. Synthesizing Robust Adversarial Examples. Assessing Generative Models via Precision and Recall. Deep Diffeomorphic Transformer Networks. Learning by Asking Questions. Gesture Recognition: Focus on the Hands.
Clipped Action Policy Gradient. Decouple Learning for Parameterized Image Operators. Adversarial Time-to-Event Modeling. Hierarchical Multi-Label Classification Networks. Convolutional Image Captioning. PacGAN: The power of two samples in generative adversarial networks. End-to-End Incremental Learning. Statistically-motivated Second-order Pooling. Excitation Backprop for RNNs. Analyzing Uncertainty in Neural Machine Translation. Learning Dynamics of Linear Denoising Autoencoders.
Density Adaptive Point Set Registration. Decoupled Parallel Backpropagation with Convergence Guarantee. Classification from Pairwise Similarity and Unlabeled Data. Fast Information-theoretic Bayesian Optimisation. Towards Realistic Predictors. A Two-Step Disentanglement Method. Conditional Prior Networks for Optical Flow. Adversarial Learning with Local Coordinate Coding. Banach Wasserstein GAN. Unsupervised holistic image generation from key local patches. Orthogonally Decoupled Variational Gaussian Processes. Disentangling Factors of Variation by Mixing Them. Neural Architecture Optimization.
Diverse and Coherent Paragraph Generation from Images.
- Strategic Management: Competitiveness and Globalization , Eighth Edition (Concepts and Cases)!
- Recommended for you;
- Fat Crystal Networks (Food Science and Technology);
Dynamic-Structured Semantic Propagation Network. The Description Length of Deep Learning models. Transfer Learning via Learning to Transfer. Deepcode: Feedback Codes via Deep Learning. Configurable Markov Decision Processes. Tracking Emerges by Colorizing Videos. Inference Suboptimality in Variational Autoencoders.
Black Box FDR. Quadrature-based features for kernel approximation.
Transferable Adversarial Perturbations. Bidirectional Retrieval Made Simple. Learning to Branch. Lifelong Learning via Progressive Distillation and Retrospection.
Parallel Bayesian Network Structure Learning. Deep Burst Denoising. Statistical Recurrent Models on Manifold valued Data. Focal Loss for Dense Object Detection. Mask R-CNN. Deep Photo Style Transfer. Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation. Attention is All you Need. Densely Connected Convolutional Networks. Deformable Convolutional Networks. Unsupervised Image-to-Image Translation Networks. Deep Feature Flow for Video Recognition.
Bayesian GAN. Pyramid Scene Parsing Network. Finding Tiny Faces. Toward Multimodal Image-to-Image Translation. FlowNet 2. Dilated Residual Networks. Inferring and Executing Programs for Visual Reasoning. Inductive Representation Learning on Large Graphs. And a Dataset of , 3D Facial Landmarks.
Video Frame Interpolation via Adaptive Convolution. Dual Path Networks. Deep Image Matting. Richer Convolutional Features for Edge Detection. Recurrent Highway Networks. Detect to Track and Track to Detect. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Universal Style Transfer via Feature Transforms. Residual Attention Network for Image Classification. One-Shot Video Object Segmentation. Feature Pyramid Networks for Object Detection.
Efficient softmax approximation for GPUs. Pixel Recursive Super Resolution. Dilated Recurrent Neural Networks. Convolutional Sequence to Sequence Learning. Prototypical Networks for Few-shot Learning. Language Modeling with Gated Convolutional Networks. Stacked Generative Adversarial Networks. Generative Face Completion. Learning From Synthetic Humans. Visual Dialog.
Inverse Compositional Spatial Transformer Networks. Input Convex Neural Networks. Oriented Response Networks. Axiomatic Attribution for Deep Networks. Gradient Episodic Memory for Continual Learning. Conditional Similarity Networks. Language Modeling with Recurrent Highway Hypernetworks. Triple Generative Adversarial Nets.
One-Sided Unsupervised Domain Mapping. Attentive Recurrent Comparators. Learning a Multi-View Stereo Machine. Bayesian Compression for Deep Learning. Adversarial Discriminative Domain Adaptation. Working hard to know your neighbor's margins: Local descriptor learning loss. Concrete Dropout. Semantic Image Synthesis via Adversarial Learning.
Hierarchical Attentive Recurrent Tracking. Deep Watershed Transform for Instance Segmentation. Associative Domain Adaptation. Wang et al. Automatic epileptic seizure detection in EEG signals using multi-domain feature extraction and nonlinear analysis. Arjunan et al. Decoding subtle forearm flexions using fractal features of surface electromyogram from single and multiple sensors. JPA ja. Yan et al. The application of mutual information-based feature selection and fuzzy LS-SVM-based classifier in motion classification. Podgorelec et al. Kumar et al. Shen et al. Precisely, we compare the statistical properties of pseudo-text generated by a neural language model with those of the real text with which the model is trained.
This finding is notable because previous language models, such as Markov models, cannot reproduce such properties, and mathematical models, which are designed to reproduce statistical laws [ 16 ] [ 17 ], are also limited in their purpose. As compared with those models, neural language models are far more advanced in satisfying the statistical laws. We find a shortcoming of neural language models, however, in that the generated pseudo-text has a limitation with respect to satisfying a third statistical property, the long-range correlation.
The analyses described in this paper contribute to our understanding of the performance of neural networks and provide guidance as to how we can improve models. We constructed a neural language model that learns from a corpus and generates a pseudo-text, and then investigated whether the model produced any statistical laws of language. Bengio et al. We construct a language model at the character level, which we denote as a stacked long short-term memory LSTM [ 22 ] model.
This model consists of three LSTM layers with units each and a softmax output layer. We treat this stacked LSTM model as a representative of neural language models. In all experiments in this article, the model was trained to minimize the cross-entropy by using an Adam optimizer with the proposed hyper-parameters [ 23 ]. The context length k was set to To avoid sample biases and hence increase the generalization performance, the dataset was shuffled during the training procedure: i.
This is a standard configuration with respect to previous research on neural language models [ 19 ] [ 20 ] [ 21 ] [ 24 ]. In the normal scheme of deep learning research, the model learns from all the samples of the training dataset once during an epoch. That is, the model learns from all the samples every epochs. Generation of a pseudo-text begins with characters in succession as context, where the character sequence exists in the original text.
The context is then shifted ahead by one character to include the latest character. This procedure is repeated to produce a pseudo-text of 2 million characters unless otherwise noted. We chose a character-level language model because word-level models have the critical problem of being unable to introduce new words during generation: by definition, they do not generate new words unless special architectures are added.
With such a model, there is a definite vocabulary size limit, thus destroying the tail of the rank-frequency distribution. Therefore, we chose a character-level language model. Note that the English datasets, consisting of the Complete Works of Shakespeare and The Wall Street Journal WSJ , were preprocessed according to [ 19 ] by making all alphabetical characters lower case and removing all non-alphabetical characters except spaces.
Consecutive spaces were also reduced to one space. The upper-left graph in Fig 1 shows the rank-frequency distribution of the Complete Works of Shakespeare, of 4,, characters, for n -grams ranging from uni-grams to 5-grams. As Zipf stated, the uni-gram distribution approximately follows a power law with an exponent of 1. The higher n -gram distributions also follow power laws but with smaller exponents. Note that intersection of the uni-gram and 2-gram distributions in the tail is typically observed for natural language.
The lower-left graph in Fig 1 shows the vocabulary growth of the Complete Works of Shakespeare. This exponent is larger than that reported in previous works, and this was due to the preprocessing, as previously mentioned. All axes in this and subsequent figures in this paper are in logarithmic scale, and the plots were generated using logarithmic bins. The model learned from 4,, characters in the Complete Works of Shakespeare, which was preprocessed as described in the main text.
The colored sets of the plots of the figures in the first row show the rank-frequency distributions of 1,2,3,4,5 grams. The figures in the second row show the vocabulary growth in red. For all graphs, the corresponding estimated exponents are indicated in the caption, and the black solid line in each vocabulary growth figure shows the fitted line.
The dashed line indicates a reference with an exponent of 1. The same applies to all other rank-frequency distribution and vocabulary growth plots in this paper. The graphs on the right side of Fig 1 show the corresponding rank-frequency distribution and vocabulary growth of the pseudo-text generated by the stacked LSTM. The rank-frequency distribution is almost identical to that of the Complete Works of Shakespeare for uni-grams and 2-grams, reproducing the original shape of the distribution.
The distributions for longer n -grams are also well reproduced.
Esteemed computer science prof dies
As for the vocabulary growth, the language model introduces new words according to a power law with a slightly larger exponent than that of the original text. This suggests a limitation on the recognition of words and the organization of n -gram sequences. These results indicate that the stacked LSTM can reproduce an n -gram structure closely resembling the original structure. The potential of the stacked LSTM is still apparent even when we change the kind of text. Xueqin, respectively, and the corresponding pseudo-texts generated by the stacked LSTM.
The WSJ text of 4,, characters was subjected to the same preprocessing as for the Complete Works of Shakespeare. To deal with the large vocabulary size of the Chinese characters, the model was trained at the byte level [ 30 ] for Hong Lou Meng, resulting in a text of 2,, bytes.
1 Neurophysiology of feedback
To measure the rank-frequency distribution and vocabulary growth at the word level, the model had to learn not only the sequence of bytes but also the splits between them. The model learned from 4,, characters. The length of the pseudo-text is 20 million characters.
The rank-frequency distributions are shown for 1,2,3,4,5,8,grams. The preprocessing procedure was the same as for the Complete Works of Shakespeare. The text was processed at the byte level with word borders. The observations made for the Complete Works of Shakespeare apply also to Figs 2 and 3. We observe power laws for both the rank-frequency distributions and the vocabulary growth. The stacked LSTM replicates the power-law behaviors well, reproducing approximately the same shapes for smaller n -grams.
The intersection of the uni-gram and 2-gram rank-frequency distributions is reproduced as well. As for the vocabulary growth, the reproduced exponents were a little larger than the original values, as seen for the case of Shakespeare. Fig 2 also highlights the high capacity of the stacked LSTM in learning with long n -grams. In the Complete Works of Shakespeare, written by a single author, long repeated n -grams hardly occur, but the WSJ dataset contains many of these. For the WSJ data, the rank-frequency distributions of 8- and grams do not obviously follow power laws, mainly because of repetition of the same expressions.
With such a corpus, the stacked LSTM can also reproduce the power-law behavior of the rank-frequency distribution of long n -grams. These results indicate that a neural language model can learn the statistical laws behind natural language, and that the stacked LSTM is especially capable of reproducing both patterns of n -grams and the properties of vocabulary growth.
We also tested language models with different architectures. With the CNN upper left , on the other hand, the shape of the rank-frequency distribution is quite different, and the exponent of the vocabulary growth is too large. The simple RNN upper right shows weaker capacity in reproducing longer n -grams, and the exponent is still too large. It starts learning obviously at the level of a monkey typing.
Fig 4 shows the rank-frequency distribution and vocabulary growth of a texts generated by the stacked LSTM without training. Each case from uni-grams to 3-grams roughly forms a power-law kind of step function. As shown by Fig 4 , monkey-typed texts can theoretically produce power-law-like behaviors in the rank-frequency distribution and vocabulary growth. Following the explanation in [ 32 ], we briefly summarize the rationale as follows.
Consider a monkey that randomly types any of n characters and the space bar. Since , the rank r c of a word of length c grows exponentially with respect to c ; i. Given that the probability of occurrence of a word of length c is , by replacing c with the rank, we obtain the rank-probability distribution as 4 where the log is taken with base n. This result shows that the probability distribution follows a power law with respect to the rank. The LSTM models therefore start learning by innately possessing a power-law feature for the rank-frequency distribution and vocabulary growth.
The learning process thus smooths the step-like function into a more continuous distribution; moreover, it decreases the exponent for vocabulary growth. Fig 5 illustrates the training progress of the language model for the Complete Works of Shakespeare. The upper-left graph shows the cross entropy of the model at different training epochs. The training successfully decreases the cross entropy and reaches a convergent state. The left-hand graphs are in logarithmic scale for the x-axes and linear scale for the y-axes.
They generally increase and become equivalent to the values of the original datasets for short n -grams or remain at smaller values for long n -grams. It roughly stops decreasing, however, at around 10 2 to10 3 epochs. The right-hand side of Fig 5 shows the rank-frequency distributions of the pseudo-texts generated at different epochs.
- Programming C# 5.0: Building Windows 8, Web, and Desktop Applications for the .NET 4.5 Framework;
- Mathematics of language learning - Persée!
- Frommers Germany 2006?
The stacked LSTM model reproduces the power-law behavior well for uni-grams and 2-grams, and partially for 3-grams, with just a single epoch upper right. Such behavior for 4-grams appears in epoch 2 middle left , and the intersection of the uni-gram and 3-gram power laws appears in epoch 7 middle right. Power-law behavior for 5-grams emerges in epoch 51 bottom left , and no further qualitative change is observed afterwards bottom right.
As training progresses, the stacked LSTM first learns short patterns uni-grams and 2-grams and then gradually acquires longer patterns 3- to 5-grams. There are no tipping points at which the neural nets drastically change their behavior, and the two power laws are both acquired at a fairly early stage of learning. Natural language has structural features other than n -grams that underlie the arrangement of words.
A representative of such features is grammar, which has been described in various ways in the linguistics domain. The structure underlying the arrangement of words has been reported to be scale-free, globally ranging across sentences and at the whole-text level. One quantification methodology for such global structure is long-range correlation. Long-range correlation describes a property by which two subsequences within a sequence remain similar even with a long distance between them.
Typically, such sequences have a power-law relationship between the distance and the similarity. This statistical property is observed for various sequences in complex systems. Various studies [ 34 — 41 ] report that natural language has long-range correlation as well. Measurement of long-range correlation is not a simple problem, as we will see, and various methods have been proposed. The mutual information at a distance s is defined as 5 where X and Y are random variables of elements in each of two subsequences at distance s. They also provide empirical evidence that a Wikipedia source from the enwik8 dataset exhibits power decay of the mutual information, and that a pseudo-text generated from Wikipedia also exhibits power decay when measured at the character level.
Related The Informational Complexity of Learning: Perspectives on Neural Networks and Generative Grammar
Copyright 2019 - All Right Reserved